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Dear NEAIR Friends and Colleagues, 

The Association’s 32 nd Annual Conference was held in beautiful Saratoga Springs, New York, a town 
known for its “health, history, and horses.” The town welcomed over 300 of our members and guests 
from November 5 through 8, 2005, providing ample opportunities for shopping in the downtown area 
surrounding the host hotel, The Saratoga, as well as relaxation and rejuvenation in the city’s historic baths 
and spas. The conference set a new record for registrations with 299 paid registrations. 

The theme for this year’s conference was “The Race for Answers: Clarity versus Information Overload. ” 
Attendees had a record number of Pre-Conference workshops from which to choose (17) as well as a 
Colloquium for Experienced practitioners. Skidmore College hosted a variety of computer-based 
workshops. Early arrivals were treated to a taste of the Saratoga racing culture during the Saturday night 
reception “A Night at the Races” - members enjoyed cheering on their favorite horses during the video 
racing as well as sporting festive racing-themed attire. The official beginning of the conference was 
marked by the Sunday evening keynote, delivered by Dr. John Lombardi, Chancellor of the University of 
Massachusetts Amherst who engaged attendees with his lively talk on “Public Relations, Accountability 
and Measurement: What Do You Want? How Do You Get It?” Dr. Joseph Moore, President of Empire 
State College, delivered a thought provoking and data-rich plenary session on Monday morning, “Good 
Data, Wrong Questions: The Role ofIR in Higher Education Policy. ” Attendees selected from 50 
concurrent sessions (including 5 invited speakers and 5 Vendor Showcases), approximately 20 poster 
sessions (including a poster on the results of the Association’s Member Survey), and numerous Special 
Interest Groups (SIGs) and Table Topics. Four Past Presidents - Jennifer Brown, Mary Ann Coughlin, 
Jim Trainer, and Fred Volkwein - closed the conference with a plenary panel reflecting on themes from 
the conference and trends in IR. 

The conference saw the implementation of new initiatives. First, NEAIR moved to a totally 
online registration process thanks to the considerable efforts of Roland Hall, Beth Simpson, and 
the AIR staff. The web registration enabled the association to accept credit card payments for 
the first time - a service that members had been requesting for several years. Finally, the 
association implemented its Best Practitioner Paper/IR Report, thought to be the first such award 
among AIR and its affiliates; presenters of contributed papers as well as workshares and posters 
were eligible to submit in this category. 

Throughout the conference, attendees who were involved in the association as committee 
members and mentors, as well as those in leadership positions, were recognized. Their efforts 
were reported on during the Annual Business Meeting at the Monday luncheon. Additionally, 
the membership passed a series of amendments to the governance structure in the Association’s 
Constitution; the amendments grew out of an ad hoc committee whose recommendations were 
approved by the Steering Committee and proposed to the membership for a formal vote. 

I would be remiss if I did not mention the outstanding commitment and contributions of your 
Saratoga Race Stewards, the 2005 Conference Team. Organizing a conference of this size 
required countless hours, attention to details they never knew existed, and a healthy sense of 
humor when things didn’t go as expected. This team contributed all of these to the effort and I 



will be forever grateful for their dedication, hard work, support and friendship. Kelli Parmley 
(SUNY New Paltz) served as Local Arrangements Chair and was assisted by Allison Walters 
(University of Delaware) and Joe Stankovich (Skidmore College). Mitch Nesler (Empire State 
College) served as Program Chair with the able assistance of Associate Program Chair Bruce 
Szelest (University at Albany) and Pre-Conference Workshop Coordinator Nancy Ludwig 
(Northeastern University). Jessica Shedd (NACUBO) served as chair of the Web Committee 
with assistance from Roland Hall (Georgetown University). Tim Walsh (Temple University) 
was Vendor Chair; Mindy Wang (Catholic University) was Evaluation Chair; Wendell Lorang 
(University at Albany) chaired the Best Paper Committee; and Gayle Fink (University System of 
Maryland) served as Publications Chair, coordinating the production of these Proceedings as 
well as our first-ever electronic repository for conference presentations and handouts. 
Additionally, many thanks go to the members of the Best Paper and Publications committees for 
their considerable efforts to “clean up” after the conference. Kelli Armstrong and Alan Sturtz 
coordinated the Mentoring/Newcomers Committee which assisted many first-time attendees as 
they navigated the conference. Many members assisted us by staffing the registration desk, 
leading dinner groups, and performing the numerous tasks that arise during a conference - many 
thanks for your willingness to help. Beth Simpson, the Association’s Administrative 
Coordinator was invaluable for keeping us all on task and overseeing the registration process and 
compilation of registration materials, as well as for serving as a trusted colleague and sounding 
board. 

Finally, I would like to take this opportunity to thank the members of the Steering Committee 
and the many NEAIR members who supported the association as well as me personally; I was 
overwhelmed by the notes of encouragement, offers of help, and quick responses to calls for 
assistance over the course of my year as President. NEAIR is full of truly talented, motivated, 
and supportive colleagues and for that I cannot express enough gratitude. 

Now I invite you to sample the contributions of our members during our “Race for Answers. ” I 
am certain you will find fresh ideas and support for the work you do in your institution. It is also 
my hope that the materials which follow will encourage you to participate in future conferences 
and that you find in them models for the best in reporting and publications. 

With warm wishes for your continued success, 



Michelle Appel 

NEAIR President 2004-2005 
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Abstract 

The annual performance funding program links an evaluation of each university’s 
performance to a percentage allocation of the Educational and General Appropriation funds 
distributed to the Pennsylvania State System of Higher Education. Performance is evaluated 
using institutional historical baselines, external benchmarks, and System Performance Targets. 
By providing financial incentives for improving performance within eight funding areas, the 
System is moving the 14 member universities in strategically desirable directions. 

Introduction 

Higher education has seen significant increases in performance-based funding since its 
introduction by the state of Tennessee in 1978. Burke and Minassians’ (2003) survey details the 
growth in performance funding — from 10 states in 1997 to 15 states in 2003 (with a high of 19 
states in 2001). States have implemented performance funding include increasing pressures to 
measure institutional performance with limited fiscal dollars (Woodley, 2005), requirements that 
universities be responsive to stakeholders (Borden and Banta, 1994; Armstrong, et. al., 2004), 
and demands for higher education institutions to improve quality, productivity, and effectiveness 
(Peterson and Augustine, 2000). 

Performance funding recognizes that additional financial resources positively impact 
institutional behavior. By targeting certain quantitative measures of institutional performance, 
additional financial resources not only reward institutions for high levels of performance, but 
also provide incentives for institutions to prioritize their efforts to achieve improved 
performance. 

Allocation of funds based upon performance tends to fall into two general performance- 
based funding categories: performance funding and performance budgeting (Burke and 
Minassians, 2003). Performance funding provides state funds directly to universities based on 



'The authors would like to recognize Christina Hendriksen, Jeff Kinsey, Sara Senko, and Melinda Tobin, 
Pennsylvania State System of Higher Education, for their contributions to the Performance Funding Program. Any 
errors are the responsibility of the authors. 
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performance on individual indicators. Performance budgeting provides the possibility of 
additional funding due to the discretion of state, coordinating and system officials who use 
university performance on individual indicators as one factor in determining performance 
allocations. 

The Pennsylvania System of Higher Education’s (PASSHE) current Performance 
Funding Program was developed with university presidents and approved by the Board of 
Governors in October 2002. The program was not initiated in response to legislation. Adoption 
of the Performance Funding Program without a mandate is common, as 53% of such programs 
are not required by legislation (Burke and Minassians, 2003). 

PASSHE’s Performance Funding Program awarded 7% of its base appropriation from the 
Commonwealth for FY 2005-2006. PASSHE’s current percentage funding level is in the high 
range of states with performance funding programs — the range is from 0.5% to about 6% with 
the average being 2% (Burke, 2005). 

The next section of this paper provides details about PASSHE’s Performance Funding 
Program. While Burke and Minassians’ (2003) survey indicated that performance funding has 
had a minimal to moderate impact on improved performance of public colleges and/or 
universities for 67% of states having a performance funding program, some early trends reported 
by PASSHE suggest their Performance Funding Program has had an important impact. 

Pennsylvania State System of Higher Education’s Performance Funding Program 

In 2000, the PASSHE Board of Governors began allocating a portion of the 
Commonwealth’s appropriation to the universities based on their performance. Based upon the 
report, System Funding Formula and Performance Funding Concept Paper for the Pennsylvania 
State System of Higher Education (July 12, 2001), the current Performance Funding Program 
was approved by the Board of Governors in October 2002. The Program utili z es eight of the 17 
measures of the System Accountability Plan - a performance reporting program used for 
accountability purposes (Armstrong, et. al., 2004; Office of the Chancellor, August 2005). 

PASSHE’s Performance Funding Program awarded $2 million in funding to 14 
universities in its first year, FY 2000-2001, and has grown to 7% or $31,174,700, of PASSHE’s 
Educational and General Fund appropriation from the Commonwealth for FY 2005-2006. The 
14 PASSHE Universities are Bloomsburg, California, Cheyney, Clarion, East Stroudsburg, 
Edinboro, Indiana, Kutztown, Lock Haven, Mansfield, Millersville, Shippensburg, Slippery 
Rock, and West Chester University. 

In 2004, the Board of Governor’s adopted PASSHE’s Strategic Plan, Leading the Way, 
resulting in the alignment of the eight performance funding measures within the Strategic Plan 
Goal Categories. Performance funding provides financial incentives for Goal improvement as 
envisioned in the Strategic Plan. Listed below are the four Goal Categories (the fifth Category, 
Public Leadership, does not include a performance funding measure) aligned with the 
performance funding measures: 
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(1) Student Achievement and Success: Second Year Persistence; Graduation - Four 
and Six Year; 

(2) University and System Excellence: Degrees Awarded - Bachelor’s; Employee 
Diversity; Faculty with Terminal Degrees; 

(3) Commonwealth Service: Degrees Awarded - Masters; Masters Cost per Full Time 
Equivalent (FTE) Student; and 

(4) Resource Development and Stewardship: Faculty Productivity; Personnel Ratio; 
Undergraduate Cost per Full Time Equivalent (FTE) Student. 

The distribution of performance funding monies parallels the methodology used to 
determine universities’ quantitative performance within the three components of the System 
Accountability Plan: Institutional Improvement, Comparative Achievement, and Performance 
Target Attainment. 

Institutional Improvement evaluates performance relative to a historical baseline. 
University performance on measures is determined by comparing actual performance to a 
historical baseline developed using the University’s and System’s historical data. Evaluation 
categories are described in Table 1. 



Table 1: Baseline Evaluation Categories 



Baseline 

Evaluation 


How well a university did in attaining their target relative to the 
projected performance baseline for the current year 


Exceeded 


Baseline is exceeded if performance is at or above the upper bound for 
measures that are expected to increase (at or below the lower bound for 
measures that are expected to decrease*) 


Met 


Baseline is met if performance is within the upper bound and equal to or 
greater than the lower bound around the target (above the lower bound and 
at or below the upper bound for measures expected to decrease*) 


Not Met 


Baseline is not met if performance is below the lower bound for measures 
that are expected to increase (above the upper bound for measures that are 

expected to decrease*) 



Note: A within sample of one standard deviation is generally used for bounds but for some measures, a standard deviation of values is generated 
by prediction methods used. 

*The inverted measures are personnel ratio and instructional costs per FTE student. 



The second evaluative component, Comparative Achievement, uses external 
standards/benchmarks to evaluate a university’s performance. Benchmarks include external 
standard comparison groups of state-wide benchmarks, national benchmarks. University selected 
peer institutions, and System-wide averages where external data are not available. The 
methodology for determining performance is shown in Table 2. 
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Table 2: Benchmark Performance Evaluation Categories 



Benchmark 

Evaluation 


How well a university performed compared to an external standard 

comparison group data 


Exceeded 


Benchmark is exceeded if performance is one standard deviation or more 
above the average level for the external standard (one standard deviation or 
more below the average level for the external standard of measures that are 

expected to decrease*) 


Met 


Benchmark is met if performance is above or equal to the average level for 
the external standard but below the average plus one standard deviation 
(below or equal to the average level for the external standard but above the 
average minus on standard deviation for measures that are expected to 

decrease*) 


Not Met 


Benchmark is not met if performance is below the average level for the 
external standard (above the average level for the external standard*) 



Note: National benchmark data for graduation and retention rates at public institutions are obtained from the Consortium for Student 
Retention Data Exchange. For national benchmarking, universities are clustered by both selectivity (as measured by average Scholastic 
Assessment Test scores for entering freshmen) and Carnegie classification. For other measures, 15 peers in the same Carnegie 
classification were selected for each university. For some measures, benchmark data is unavailable — in those cases a System average is 
used as the benchmark. 

*The inverted measures are personnel ratio and instructional costs per FTE student. 



Starting with the 2004-2005 reporting year, university performance was evaluated 
annually relative to a university’s progress in attaining PASSHE’s System Performance Targets 
(SPT) as the third performance evaluation category. Performance Target Attainment. The SPTs 
are System average goals that are unchanged until 2009 and set for all measures and sub- 
measures (Armstrong, et. al., 2004). Unlike the previous two evaluation categories, the SPTs are 
the same for all universities. The methodology is consistent with benchmarking and institutional 
baseline (see Table 3). 



Table 3: System Performance Target Evaluation Categories 



System 

Performance 

Target 

Evaluation 


How well a university did as compared to the System Performance Target 

for the current year 


Exceeded 


System Performance Target is exceeded if performance is at or above the upper 
bound for measures that are expected to increase (at or below the lower bound for 
measures that are expected to decrease*) 


Met 


System Performance Target is met if performance is below the upper bound and 
equal to or greater than the lower bound around the target (above the lower bound 
and at or below the upper bound for measures expected to decrease*) 


Not Met 


System Performance Target is not met if performance is below the lower bound 
for measures that are expected to increase (above the upper bound for measures 

that are expected to decrease*) 



*The inverted measures are personnel ratio and instructional costs per FTE student. 



Performance funding provides resources to universities based on how well they compared 
to their own baselines, benchmarks, and the SPTs, and how many other PASSHE universities 
were in the same or a higher performance bracket for each measure and sub-measure. 
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Universities who “met” or “exceeded” baselines, benchmarks, or SPTs for any of the eight 
performance funding measures receive additional funding. 



Six of the eight performance funding measures have sub-measures for a total of 23 sub- 
measures evaluated for baselines (see Table 4). Because benchmarks and SPTs evaluations are 
limited to percentage and ratio comparisons, 14 sub-measures are used for these performance 
evaluation categories. 



Table 4: Performance Funding Measures and Sub-Measures for Evaluation 



Number 


Measure 


Sub-Measures 


Baselines 


Benchmarks 


System 

Performance 

Targets 


i 


Degrees Awarded 


Number - Bachelor's 


X 






Degree to Enrollment Ratio - Bachelor's 


X 


X 


X 


Number - Masters 


X 






Degree to Enrollment Ratio - Masters 


X 


X 


X 


2 


Second Year Persistence 


Students Persisting - Overall 


X 






Retention Rate - Overall 


X 


X 


X 


Students Persisting - Black and Hispanic combined 


X 






Retention Rate - Black and Hispanic combined 


X 


X 


X 


3 


Graduation Rates 


Number of Students who Graduated in Four Years - 
Overall 


X 






Percent of Students who Graduated in Four Years - 
Overall 


X 


X 


X 


Number of Students who Graduated in Four Years - 
Black and Hispanic combined 


X 






Percent of Students who Graduated in Four Years - 
Black and Hispanic combined 


X 


X 


X 


Number of Students who Graduated in Six Years - 
Overall 


X 






Percent of Students who Graduated in Six Years - 
Overall 


X 


X 


X 


Number of Students who Graduated in Six Years - Black 
and Hispanic combined 


X 






Percent of Students who Graduated in Six Years - Black 
and Hispanic combined 


X 


X 


X 


4 


Faculty Productivity 


Total Credits per FTE Instructional Faculty 


X 


X 


X 


5 


Employee Diversity 


Number of Minority Faculty 


X 






Percent of Faculty who are Minority 


X 


X 


X 


6 


Personnel Ratio 


Total Personnel Compensation as a Percent of Total 
Expenditures and Transfers 


X 


X 


X 


7 


Instructional Cost 


Masters Cost per FTE Student 


X 


X 


X 


Undergraduate Cost per FTE Student 


X 


X 


X 


8 


Faculty Terminal Degrees 


Percent of Full-time Tenured or Tenure Track 
Instructional Faculty 


X 


X 


X 



Note: White and Hispanic students are used instead of Black and Hispanic for Cheyney University. 



Table 5 briefly lists the description of the data for all performance funding measures and 
sub-measures (see Office of the Chancellor, August 2005, for greater detail). Data definitions 
and criteria are regularly reviewed for comparability. As a means of ensuring accountability and 
equity, the university self-reported statistics are audited by PASSHE’s Internal Review Group on 
a cyclical basis to ensure the accuracy of reported results. 
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Table 5: Performance Funding Measures and Sub-Measures: Descriptions 



Measure 


Calculation of Measure 


Date(s) of Baseline Measure 


Date(s) of External 
Benchmark/System Average and 
Peer Group 


System Performance Targets 


Degrees 

Awarded 


# of degrees/# fall FTE for same level & year 
(bachelor 1 s/master 1 s) . 


Bachelor’s: 2004-2005 degrees 
awarded and the average of fall 
1999-2001 undergraduate 
enrollments; Masters: 2004-2005 
degrees awarded and the average 
of fall 2003 and 2004 graduate 
enrollments 


Bachelor's: 2003-2004 degrees 
awarded and the average of fall 
1998-2000 undergraduate 
enrollments; Institutional Peer 
(IPEDS) public 


Degree to Enrollment Ratio - 
Bachelor's (21.50%) and Master's 
(67.00%) 


Second Year 
Persistence 
Rate 


# of cohort students returning second year 
(overall and by combined black and hispanic 
ethnicity)/ # of total cohort students; includes 
full-time first- time degree- seeking freshmen. 


Fall 2003 cohort 


Fall 2003 cohort; National Cluster 
(CSRDE) public 


Retention Rate - Overall (79.00%) 
and Black and Hispanic Combined 
(79.00%) 


Four and Six 
Year 

Graduation 

Rates 


# of cohort students graduated/ # of total 
cohort students (within 4 and 6 years). 
Overall and by combined black and hispanic 
ethnicity; includes full-time first-time degree- 
seeking freshmen 


Fall 2000 cohort for 4 years; Fall 
1998 cohort for 6 years 


Fall 1998 cohort for 6 years, Fall 
2000 cohort for 4 years; National 
Cluster (CSRDE) public 


Percent of Students who 
Graduated in Four Years and Six 
Years- Overall (30.00% and 
55.00%) and Black and Hispanic 
Combined (30.00% and 55.00%) 


Faculty 

Productivity 


# of annualized student credit hours/# of 
instructional FTE faculty 


July 1, 2003 to June 30, 2004 


July 1, 2003 to June 30, 2004; 
System Average 


Total Credits per FTE Instructional 
Faculty (565.00) 


Employee 

Diversity 


# of FT minority faculty/# of FT faculty 


Fall 2003 


Fall Staff 2003; Institutional Peer 
(IPEDS) public 


Percent of Faculty who are 
Minority (15.00%) 


Personnel 

Ratio 


Total E&G personnel costs/Total 
Expenditures and Transfers 


2003-04 


2003-04; Institutional Peer 
(IPEDS) public 


Personnel Ratio (73.00%) 


Instructional 

Costs 


Total instructional costs of UG FTES/# of 
UG FTES (and masters) 


2003-04 


2003-04; System Average 


Undergraduate and Master's Cost 
per FTE Student (3.50% and 
7.00%) 


Faculty 

Terminal 

Degrees 


# of FT permanent tenured and tenure track 
instructional faculty with terminal degrees/# 
of FT permanent tenured-tenure track 
instructional faculty (as of October 31.) 


Fall 2004 


Fall 2004; System Average 


Percent of Full-time Tenured or 
Tenure Track Instructional Faculty 
(90.00%) 



Note: Faculty Productivity and Instructional Cost performance funding measures do not include doctoral/first professional instruction data. 



Performance Funding Program: Final Distribution of Funding Dollars by University 

The Performance Funding Program provides additional financial resources to those 
universities who perform at relatively high levels and rewards those who focus their efforts 
on achieving improved performance relative to the other universities. The amount of the 
award for FY 2005-2006 is based on performance evaluation and divided equally by the 
three performance categories ($3 1,174, 700/3=$ 10, 39 1,567). Within the three performance 
categories, dollars are divided evenly across the eight measures ($10,391,567/8= 

$1,298,946). If a measure has sub-measures, the dollars for that particular measure are 
divided equally across sub-measures. For example from Table 4, Second Year Persistence 
has four sub-measures, where total dollars for each sub-measure equal $324,736 
($1,298,946/4) for the baseline performance category. 

In the aggregate, total dollars awarded in the “exceeded” performance category will be 
greater than those awarded in the “met” category. To ensure this outcome, the dollars for all 
measure/sub-measures are distributed such that universities with performance in the “exceeded” 
category receive an award that is always greater per FTE student than that of those with 
performance in the “met” category. 
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Table 6 provides an example of the performance funding allocation for the sub-measure 
Second Year Persistence, Students Persisting - Overall. For this sub-measure, two universities 
were in the “exceeded” baseline category; 12 universities were in the “met” baseline category, 
and zero universities were “not met”. Initially for each baseline, total dollars are allocated to 
each sub-measure, where the university receives three shares for “exceeded” and one share for 
“met”. Thus, six shares are allocated to the “exceeded” category (3 x 2) for a 33% distribution 
of dollars (6/18) or $108,245. Twelve shares are allocated to the “met” category (1 x 12) for a 
67% distribution of dollars (12/18) or $216,491. Note: No dollars are awarded for a 
performance evaluation of “not met”. 

The total university performance award is adjusted to university size as measured by total 
FTE students at the appropriate undergraduate and graduate levels. Table 6 shows that for the 
two universities in the “exceeded” category for Students Persisting - Overall, the sum of their 
total full time equivalent student (FTES) for fall 2004 is 13,661.6 and for the “met” category, 
79,772.8. By dividing dollars allocated to the “exceeded” category over the total FTES within 
that category, dollars per FTES for each performance category is calculated (i.e., $7.92 for 
“exceeded” and $2.71 for “met”). 



Table 6: Students Persisting - Overall ($324,736 Allocation) 





Total 


Exceeded 


Met 


Not Met 


Number of Universities in 
each Performance Category 


14 


2 


12 


0 


Pro-Ration 


3 to 1 


3x2 = 6 


1 x 12= 12 


0 


Total Shares (%) 


18 (100%) 


6 (33.33%) 


12 (66.67%) 


0 (0%) 


Allocation 


$324,736 


$108,245 


$216,491 


$0 


Total FTES in Performance 
Category (Fall 2004) 


93,434.4 


13,661.6 


79,772.8 


- 


Dollars per FTES 


$10.63 


$7.92 


$2.71 


- 



Note: Figures may not total exactly due to rounding. 



The initial allocation of performance funding dollars to PAS SHE’s 14 universities 
for Second Year Persistence: Students Persisting — Overall is calculated as the dollars per 
FTES by the total FTES for each university for the “exceeded” and “met” categories. This 
process is repeated for all performance funding measures and sub-measures. 

To determine the final allocation of funding dollars, two adjustments have been 
established. First, if a SPT is “exceeded”, and the baseline was initially evaluated as “not 
met”, the baseline evaluation is changed to “met”. Second, an adjustment is made for 
measures or sub-measures where no university performance falls in either the “exceeded” or 
“met” categories. The funding pool for those measures/sub-measures is evenly redistributed 
to the baseline, benchmark, or SPT portions of the other measures/sub-measures within 
“exceeded” categories for universities. Table 7 provides the final distribution of 
performance funding dollars after the adjustments were implemented for Fiscal Year 2005- 
2006. 
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Table 7: Final Distribution of Performance Funding Dollars 



University 


Baseline 


Benchmark 


System 

Performance 

Target 


Total 


Bloomsburg 


$714,432 


$871,555 


$1,092,095 


$2,678,082 


California 


$620,993 


$987,924 


$592,869 


$2,201,786 


Cheyney 


$152,764 


$144,805 


$496,938 


$794,507 


Clarion 


$593,693 


$525,766 


$421,173 


$1,540,632 


East 

Stroudsburg 


$411,720 


$465,024 


$341,949 


$1,218,693 


Edinboro 


$802,813 


$251,637 


$145,858 


$1,200,308 


Indiana 


$1,682,359 


$993,863 


$766,253 


$3,442,475 


Kutztown 


$792,033 


$548,897 


$338,715 


$1,679,645 


Lock Haven 


$430,660 


$543,862 


$147,190 


$1,121,712 


Mansfield 


$426,578 


$466,863 


$130,133 


$1,023,574 


Millersville 


$701,104 


$1,263,372 


$2,597,804 


$4,562,280 


Shippensburg 


$569,596 


$972,432 


$1,008,197 


$2,550,225 


Slippery Rock 


$1,142,749 


$783,886 


$796,865 


$2,723,500 


West Chester 


$1,350,073 


$1,571,681 


$1,515,527 


$4,437,281 


Total 


$10,391,567 


$10,391,567 


$10,391,566 


$31,174,700 



Note: Figures may not total exactly due to rounding. 



Conclusion 

The principal rationale for performance funding is that it provides financial 
incentives that encourage universities to achieve desirable goals (Burke, 2002). 

Nevertheless, Burke and Minassians’ (2003) survey of public colleges and/or universities for 
2003 reported that 67% of performance funding programs had a minimal to moderate 
impact, including Pennsylvania, on improved performance. 

While recognizing the data are of a limited time period and scope, and other factors 
may impact the performance funding measures (see Burke, 2005), generally PASSHE is 
moving in desirable strategic directions as indicated by the 2000-2005 System total trends 
(see summary below). The performance funding sub-measure trends (percent or ratio only) 
are listed below within their Strategic Plan Goal Categories (see link for broader discussion 
of the trends, http://www.passhe.edU/content/7/performance): 

(1) Student Achievement and Success: Second Year Persistence - Overall (1.82%) and 
Black and Hispanic combined (3.91%); Graduation - Four Year Overall (0.36%) and 
Black and Hispanic combined (3.24%); Graduation - Six Year Overall (0.70%) and 
Black and Hispanic combined (9.78%); 

(2) University and System Excellence: Degrees Awarded - Bachelor’s (1.96%); 
Employee Diversity 0.27%); Faculty with Terminal Degrees (4.00%); 

(3) Commonwealth Service: Degrees Awarded - Masters (-1.89%); Masters Cost per 
FTE Student (8.56%); and 
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(4) Resource Development and Stewardship: Faculty Productivity (7.47%); Personnel 

Ratio (1.78%); Undergraduate Cost per FTE Student (5.82%). 

All performance funding sub-measures except for Degree to Enrollment Ratio - 
Masters; Personnel Ratio; and Instructional Cost per FTE Student have improved. For 
Degrees Award - Masters, the number of Masters degrees increased by 17.7% over the time 
period, but the cohort increased relatively faster by 22%, resulting in a decline. For the 
Personnel Ratio and Instructional Cost per FTE Student, the costs have risen but at a slower 
rate than the 13.8% change in inflation (consumer price index - all urban consumers) over 
the same time period resulting in real declines in cost. 

The Performance Funding Program has provided additional resources as a method to 
strengthen the System Goal Categories as presented within PASSHE’s Strategic Plan. When 
performance funding dollars become increasingly important to an institution’s overall revenue 
stream, universities devote both time and resources to implement strategies that will improve 
performance in each of the funded areas. 
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USING FACTOR ANALYSIS TO CLARIFY OPERATIONAL CONSTRUCTS 
FOR MEASURING MISSION PERCEPTION 



Ellen M. Boylan, Ph.D. 

Associate Director of Institutional Research 
Office of Planning and Institutional Research 
Marywood University 



Introduction 

This paper describes a mixed methods strategy of inquiry for developing and testing 
factors that represent constructs of institutional mission. The primary objective of this research 
was to develop an instrument that measures a college student’s perception of mission, in order to 
provide some evidence of mission effectiveness to address accreditation standards. A second 
objective, once an administration of the instrument was conducted and an adequate data set 
obtained, was to investigate the instrument’s internal reliability. Third, if the instrument was 
indeed found reliable, the next objective was to uncover the presence within of any constructs or 
subscales of items. Last, if operational constructs were present within the instrument, the final 
objective was to observe relationships among them using correlation analysis. 

The study was designed to address the following research questions: 

• Are the Research Mission Questions valid and reliable measures of student 
perception of institutional mission? 

• Are there factors present in the Research Mission Questions? 

• If there are factors, what is the relationship among them? 

Review of the Literature 

The pace of assessment activity in United States colleges and universities has been 
accelerating rapidly over the past two decades. Actors in the growing accountability movement 
search avidly for robust indicators of higher education performance (Ewell, 1998). The need is 
particularly acute relative to accreditation standards on mission, where performance indicators 
representing mission constructs are largely absent. 

Chickering (1993) points out the importance for an institution to have clearly articulated 
objectives for student learning and development. Ideally, the objectives pervade the institution’s 
programs and climate, and are widely shared and emphasized in oral and written communication. 
A strong sense of college mission that unifies the educational experience of students can reduce 
ambiguity and define purpose for students and others in the institution. Some of the objectives, 
or constructs, present in the mission statements of institutions participating in this research are 
shown in Appendix A. 
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The relationship between institutional quality, institutional characteristics like student 
backgrounds, and student success have been examined, but the relationship of institutional 
mission to student engagement and education outcomes has not been fully explored (Pike, Kuh, 
& Gonyea, 2002). The U.S. News and World Report’s America ’s Best Colleges 2005 defines an 
institution’s mission according to its Carnegie Classification (2004, p.80), a conventionally 
accepted taxonomy of American colleges and universities originally devised by the Carnegie 
Foundation. In this research, however, “mission” is operationally defined as the cluster of 
institutional goals found in the mission statement of a college or university. The relation between 
an institution’s mission goals and student engagement and learning environment are explored. 

Research by Pike, Kuh, and Gonyea (2002) on institutional mission concludes that 
mission constructs can be measured. The methodology used by the National Survey of Student 
Engagement (NSSE) for producing benchmarks, or scales, of “effective educational practice” 
(Kuh, 2001, p. 13) is mirrored by this research in the way factor scales were produced. 

The purpose of the factor analytic method used here was to evaluate score validity once 
the measure had been developed, and focus on whether the scores measure “the correct 
something” (Thompson, 2004, p. 4), and are valid. Tests used in this research are meant to 
clarify operational constructs for an area, in this case, the measurement of student perception of 
institutional mission. The scores, or outcomes, of those measures can then be a means for 
evaluating students’ perceptions of mission-related curricular and co-curricular programs on 
campus. 

Thompson and Daniel (1996) believe that factorial validity of a test is, “given by its 
loadings in meaningful, common, reference factors” (p. 197). The Rotated Factor Matrix 
(Appendix B) shows how the test given in this research performed against that standard. 

Study Limitations . This study was based on research conducted at a specific consortium 
of schools. Although the results may not be generalizable to other institutions, the methodology 
may be adapted to explorations of institutional mission at other colleges. 

Data Source 

The study is based on data from 3,857 first-year and senior students at fourteen schools 
participating in the NSSE 2004 Catholic College Consortium. The NSSE instrument, the College 
Student Report (CSR), was administered to the consortium sample with an added 20-item set of 
Research Mission Questions. Respondents were classified as first year students (N=2,000) or 
seniors (N=l,827) by consortium schools, and there were 1,007 males and 2,820 females. 
Characteristics of consortium schools are noted in Table 1. 

Methodology 

The overall process of developing and validating the instrument is as follows: start with a 
draft of the question items based on constructs found in participating institutions’ mission 
statements (Appendix A), circulate the draft question items to participating institutions for 
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feedback, revise question items and obtain feedback from experts in instrument development, 
administer the instrument to potential subjects, and make final revisions. Once the instrument has 
been administered and data collected, check internal reliability, examine the item-total 
correlation matrix and delete items to produce the most reliable set of items, and conduct a factor 
analysis to explore for the presence within of reliable subscales. 



Table 1 

Institution Characteristics 



Institution 


Catholic % 


Urban/Suburban 


mu 


IM1BB 


Small 










G 


64 


S 


E 


s 


N 


42 


u 


M 


s 


Medium 










A 


36 


u 


w 


s 


B 


51 


s 


M 


s 


C 


47 


u 


M 


s 


D 


51 


s 


M 


s 


E 


60 


s 


M 


M 


F 


35 


s 


M 


M 


I 


93 


s 


M 


s 


J 


61 


s 


E 


s 


K 


68 


s 


E 


s 


L 


36 


s 


M 


s 


M 


42 


u 


M 


s 


Large 










H 


54 


u 


E 


s 



*Source: America’s Best Colleges 2005, U.S. News and World Report. 



To develop the Research Mission Questions for this research, the set of questions used 
two years prior by a different NSSE Catholic Consortium was obtained and assessed for its 
suitability. A few items were retained, but most items were developed specifically for this study 
using a qualitative research technique. The technique involved first obtaining the mission 
statements of the Consortium schools and probing them for shared goals and objectives related to 
mission. Common items were plucked out and distilled into distinct concepts. Questions that 
addressed these concepts were then drafted. Next, draft questions were distributed to the key 
contacts at the Consortium schools, usually the director of institutional research. This individual 
reviewed them, shared them with other administrators in some cases, and returned the draft 
questions with feedback and suggestions for revision. This process of review occurred in a cycle 
three times until the final questions were eventually approved by participating institutions. A 
demographic question on religious affiliation was also developed and included, since this data 
was important to Consortium participants. 

The next step in the process of developing the instrument was to check the face validity 
of the questions. The question items were shown to representatives from the Student Life area at 
the college of the Consortium Administrator (researcher) to obtain feedback on terms and 
concepts used, for example, whether “charism” or “heritage” was a better tenn to describe the 



Page 14 






















contribution of the institution’s founders. Revisions were made, and the questionnaire was then 
given to a group of 10 potential subjects from the undergraduate student body at the Consortium 
Administrator’s college, either individually or in small groups. Shortly after taking the 
questionnaire, they were asked to share comments and suggestions for clarification or 
improvement of the questionnaire. For example, one subject suggested asking whether students 
felt free to express their individual spirituality on campus, so that was added. 

In the next step, a team of experts in instrument development at NSSE was consulted. The 
team vetted the questions for content and construction, and made important suggestions about the 
Likert-scale response method and overall fonnatting. In all, 20 question items were developed. 
Some example items are: 

• Ethical and spiritual development of students is an important part of the mission at this 

institution 

• The faculty, staff, and students here are respectful of people of different races, cultures 

• How important is it to you that you accomplish the following objectives? 

1) Volunteering in community service 2) Raising a family 

Last, the questionnaire was once again distributed to Consortium schools and finally approved. 

Results 

Once the instrument was administered and data obtained, the instrument was tested using 
reliability analysis and factor analysis to answer the following questions: Are the Research 
Mission Questions valid and reliable measures of student perception of institutional mission? Are 
there factors present in the Research Mission Questions? If there are factors, what is the 
relationship among them? 

To address the first question about validity and reliability, a Cronbach’s alpha test was 
conducted on the Research Mission Questions. Items with low inter-item correlation, such as 
“becoming an authority in my field,” and “raising a family” were removed. By repeating the 
steps of analysis and removal of items with low item-total correlations (<.32), the most reliable 
set of items was produced. A reliability analysis was conducted on the remaining 17 items, 
producing a Cronbach’s alpha of .880 for the revised set of items. In the interest of clarity and 
parsimony, the 17 items were renamed the Mission Perception Inventory (MPI). 

After reliability analysis, there were three important decisions to weigh before beginning 
the process of defining factors. First, the number of possible factors that could be produced had 
to be estimated and analyzed for reliability. Second, a determination was needed as to whether 
the underlying factors were correlated or uncorrelated (orthogonal). Third, the different types of 
testing options available within factor analysis needed to be assessed and the most elegant one 
selected. The decision was made that underlying factors were likely to be correlated. 

A data reduction was needed at this juncture in order to explore the presence of a factor 
or factors in the MPI survey data set. Two factors are generated when Eigenvalues are not 
specified. Hannan (1976) advises using the basic Jacobi method of, “diagonalization of the 
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matrix R by performing a sequence of orthogonal transformations on it” (p. 145), which can be 
achieved in the SPSS program by setting the Eigenvalue at 1.0 or greater. 

Subsequently, a factor correlation analysis was performed specifying an Eigenvalue of 
1 .0 or greater and requesting four factors. The correlated factors solution to a Varimax rotation 
(Maximum Likelihood) is displayed in the Rotated Factor Matrix (Appendix B) where the values 
of R show in columns. Values of R indicate the reliability of the relationships between pairs of 
variables, and any values of R greater than .32 indicates a satisfactory relationship, or factor 
loading by items (Tabachnick, 1996). Thus, four factors are evident in the solution. The four 
factors were named Sense of Mission (8 items); Respect for Diversity (5 items); Individual 
Actions (2 items); and Religious Practice (2 items). 

Figure 1 : 

Derivation of Mission Perception Inventory (MPI) and subscales from Research Mission Questions 

Sense 

of 






Practice 
(2 items) 

To assure thoroughness of inquiry, factor scores were computed on the subscales and 
compared with the subscale scores. The comparison showed it was not necessary to employ 
factor scores for subscales in subsequent analyses, primarily because the factor score alters the 
basic metric of the scales and would affect their comparability with the overall Mission 
Perception Inventory (MPI) scale. The subscale means must be used in concert with the MPI 
scale means in subsequent analyses because they closely relate to the concepts found in 
institutional missions. 

The reliability and correlations for subscales within the MPI scale are presented in Table 
2. The Cronbach’s alpha test indicates that the two subscales, Sense of Mission (a=.87) and 
Respect for Diversity (a=.85), are very reliable and internally consistent. 



Research 
Mission 
Questions 
(20 items) 



Mission 

Perception 

Inventory 

(MPI) 

( 1 7 items) 





Mission 
(8 items) 



Respect for 
Diversity 
(5 items) 



Individual 
Actions 
(2 items) 



Religious 



Page 16 








Table 2 



Rel 


liability and Correlation of Subscales 






Correlation 


Subscale 


Cronbach a 


Sense of Mission 


Respect for Diversity 


Sense of Mission 


.867 


1.000 




Respect for Diversity 


.854 


.581 ** 


1.000 


Individual Actions 


.674 


.292 ** 


.166 ** 


Religious Practice 


.624 


.208 ** 


.089 ** 



** 

Correlation is significant at the 0.01 level (2-tailed). N=3,827. 



To address the third question about the relationship among factors, a correlation analysis 
was performed. As seen in Table 2, the correlation between the subscales indicates a significant 
relationship between Sense of Mission and Respect for Diversity (r=.581,/?<.01, df= 3,825). 
Therefore, it was concluded that the MPI subscales, Sense of Mission and Respect for Diversity, 
are significantly correlated with each other. 

The scales most useful for further analysis are Sense of Mission and Respect for 
Diversity. Individual Actions and Religious Practice each have just two variables in the scale. 
Also, the two variables in Religious Practice are scaled dichotomously, unlike the continuous 
scales in the others. 

Discussion 

To summarize, the reliability and validity of the Research Mission Questions were tested 
and found to be reliable and valid measures of student perception of institutional mission. 
However, three items in the Research Mission Question set were found to be poorly correlated, 
so they were removed to produce a 17-item set of questions called the Mission Perception 
Inventory (MPI) (a=.88). Factor analysis of the MPI then produced four reliable subscales called 
Sense of Mission, Respect for Diversity, Individual Actions, and Religious Practice. Because the 
Individual Actions and Religious Practice scales each have just two items, only Sense of Mission 
(a=.87) and Respect for Diversity (a=.85), with more items and higher reliability, would be most 
useful to any subsequent analyses that might be conducted on the data. 

Conclusions and Implications 

The four subscales produced by analysis indicate the presence of those concepts in 
students’ educational experiences at Consortium colleges. Also, the reliability of subscales 
indicates that the Mission Perception Inventory (MPI) can be employed with confidence as a 
measure of student mission perception in subsequent studies. The high reliability of the Sense of 
Mission and Respect for Diversity scales indicates, as well, that these scales can be used alone or 
in concert with other instruments to satisfactorily measure mission perception. 
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Appendix A 



Mission Statement Concepts - NSSE 2004 Catholic College Consortium 


Professional 

Development 

P 


Leadership 

L 


Academic 

Excellence 

A 


Community/ 

Service 

C 


Spiritual 

Development 

E 


Diversity 

D 






The joy of learning 
is faithfully 
nurtured. ..power of 
learning. 




Higher morals, 
and social 
conscience are 
valuable 




develop. ..personal 
competencies 




Collaborative 

environment 


Commitment to 
build a just and 
peaceful society 


Develop moral 
character 






Promotes 
leadership to 
local and 
boarded 
communities 




Promotes service 
to local and 
boarded 
communities 


Emphasizes on 
the growth of the 
human person 






development of 
leadership 


community of 
learning 


spirit of service 


promotes just and 
ethical relations; 
values 


diverse ages, 
backgrounds, and 
cultures 


preparation for 
responsible living 
and meaningful 
work 




Responsible 
academic freedom; 
provides intellectual 
preparation 




Promote moral, 
spiritual, religious 
vales to students; 
moral prep 




Imagination and 
spirit 


intellect and 
spirit 


academic 

excellence 


Service and 

leadership 

community 


Imagination and 
Spirit 






Prepare for 
leadership 


Develop 

intellectually 


Lives of service 


Develop spirit & 
heart; values- 
based education 




Prepare for 

professional 

careers 


Intellect and 
spirit 


commitment to 
serving others 
through a liberal 
arts education 


Service 

opportunities in 
the community 




diversity 


professional 

disciplines 


education 

empowers 


calls upon students 
to reach their full 
potential... lifelong 
learning 


leaders in service 


spiritual, ethical, 
and religious 
values 


live responsibly in a 
diverse and 
interdependent world 


Career preparation 


Responsibility for 
decisions 


Intellectual inquiry 


Commitment to 
service 


personal growth 


diversity 
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Appendix B 



Rotated Factor Matrix - Mission Perception Inventory 



Factor j 




1 


2 


3 


4 




Sense of 


Respect for 


Individual 


Religious 




Mission 


Diversity 


Actions 


Practice 


Social and personal development of students is an important 
part of the mission at this institution. 


.719 


.266 


.129 


.049 


Ethical and spiritual development of students is an important 
part of the mission at this institution. 


.710 


.181 


.066 


.169 


This institution offers opportunities for volunteering and 
community service. 


.674 


.141 


.149 


.030 


This institution offers opportunities for developing leadership 
skills. 


.656 


.271 


.160 


.000 


At this institution, there are opportunities for students to 
strengthen their religious commitment. 


.604 


.296 


.029 


.131 


The heritage of the founding religious community of this 
institution is evident here. 


.601 


.201 


.032 


.078 


The mission of this institution is widely understood by 
students. 


.515 


.189 


.089 


.053 


The professors at this institution discuss the ethical 
implications of what is being studied. 


.477 


.329 


.111 


.141 


The faculty, staff, and students here are respectful of people 
of different religions. 


.290 


.786 


.029 


.064 


The faculty, staff, and students here are respectful of people 
of different races and cultures. 


.361 


.747 


.012 


.104 


Students feel free to express their individual spirituality here. 


.289 


.656 


.064 


.051 


People of different sexual orientations are accepted socially 
here. 


.119 


.621 


.070 


-.081 


The environment here encourages students to develop an 
appreciation of diversity. 


.293 


.616 


.067 


-.005 


How important is it to you that you accomplish the following 
objectives? Volunteering in community service. 


.185 


.043 


.693 


.184 


How important is it to you that you accomplish the following 
objectives? Influencing social values. 


.106 


.081 


.649 


.118 


Within the past week, have you participated in a religious 
service? 


.103 


.001 


.111 


.660 


Within the past week, have you spent time in private prayer 
or meditation? 


.099 


.024 


.153 


.619 



Page 20 







OPTIMIZING THE USE OF RETENTION RESEARCH IN ASSESSMENT 



Anne Marie Delaney 
Director of Institutional Research 
Babson College 

Introduction. The purpose of this paper is to demonstrate how to design, analyze and 
present the results of retention studies to achieve optimum relevance and use in institutional 
assessment. The paper is based on a longitudinal retention study of entering freshmen at a 
selective, private college in the northeast. The study was designed to answer two major 
assessment related questions. 

• How successful are admission policies in selecting students who will 
achieve a high level of academic performance and graduate within six years? 

• Controlling for input characteristics, how well do college experiences predict 
students’ academic perfonnance and satisfaction with their education? 

Results provide a basis for assessing the effectiveness of admission policies and the impact of 
college experiences on graduates’ academic perfonnance and satisfaction. 

Literature Review. Research on student persistence in college has assumed greater 
importance in today’s competitive environment (Peltier, Laden & Matranga, 1999). This is 
particularly true with the growing demand for accountability and the increased focus on 
assessment in the accreditation of higher education institutions. 

In Assessment for Excellence . Astin (1993) proposes the Input-Environment-Output (I-E-O) 
model with the creation of a longitudinal retention file as the best long-term solution for 
assessment. This model is based on the assumption that one needs infonnation about the 
characteristics of incoming students (inputs) in order to evaluate the impact of educational 
programs and experiences (environment) on outcomes. The design of this retention study 
reflects the principles of Astin’ s model. Demographic variables (gender and citizenship) and 
admission characteristics (SAT scores and the admission rating) represent inputs. Students’ self- 
reported academic and social college experiences comprise the environment while graduation 
status and cumulative averages constitute the outcomes for the model. 

Environmental variables in this study are based primarily on senior survey results regarding 
student behaviors, self-reported gain and satisfaction. These data constitute indirect measures for 
assessment. Allen (2004) describes indirect techniques as reports about learning rather than 
direct demonstrations of learning. Ewell and Jones (1993) identified three types of indirect 
indicators for assessment: institutional requirements, instructional good practice, and student 
behaviours and self-reported gains. A rationale for using indirect measures is that they yield 
information with which to make sense of summative assessment findings and potentially provide 
clear policy leverage for action. 
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Utilizing senior survey results in the retention study provides an opportunity to enhance 
students’ voice in assessment. Ewell (1983) proposed that by using student satisfaction 
assessment results to shape institutional reform, institutions are validating the importance of the 
students’ “voice”. Echoing this view, Lingrell (1992) observed that since the ultimate 
beneficiary of assessment is the student it only seems appropriate to ask the student what he or 
she thi nk s. He also identified the senior survey as an ideal vehicle for incorporating students' 
voice. Cheng’s (2001) research provides an excellent example of how an externally developed 
senior survey may be used to construct a model for producing outcome measures in an 
institution's assessment effort. 

As Cheng and Tam (1997) have observed, higher education is increasingly recognizing that it 
is a service industry and is placing more emphasis on meeting students’ expectations and needs. 
In this context, student satisfaction is a crucial determinant of success. “Student satisfaction 
results when actual performance meets or exceeds the student’s expectations” (Elliott & Healy, 
2001, p.2). The present study includes student satisfaction in the retention/assessment model. 

Methodology. The study is based on a six-year longitudinal file for 400 first-time, full-time 
freshmen who entered college in the fall of 1998. Data were extracted from the administrative 
computer system files and merged with results from the Class of 2002 senior survey. Bivariate 
and multivariate techniques were utilized in the data analyses. Chi-Square analyses were used to 
examine relationships between demographic variables, admission characteristics and retention. 
Correlation analyses, t tests and analyses of variance were used to study the relationships 
between student characteristics, admission criteria and academic performance. Regression 
analyses were employed to predict students’ academic performance and satisfaction with college. 

Study Limitations . The study is based on research conducted at a single institution, which 
potentially restricts the variance for analysis and the generalizability of the findings. Further, the 
research was limited by a lack of information on: input characteristics; non-graduates’ reasons 
for leaving; and student involvement in student life as well as by the absence of an integrated 
survey/administrative data base. The final report from the study included several 
recommendations designed to address these issues. 

Results 

Graduation Rates. The overall graduation rate for the fall 1998 entering freshman class was 
85.5 percent. Analysis by demographic characteristics revealed a significantly higher graduation 
rate of 93 percent among females, compared with 82 percent among males (X~=T 1.51, p < .05). 
Though not statistically significant, the graduation rate was higher at 87 percent among 
international students, compared with 85 percent among U.S. citizens, and the graduation rate of 
88 percent among financial aid recipients, exceeded the rate of 82 percent among non-financial 
aid recipients. 

Analyses of the relationship between admission characteristics and graduation rates revealed 
no statistically significant differences by SAT scores. However, results did show that all non- 
graduates with SAT scores between 700 and 800 left voluntarily, while all non-graduates with 
SAT Math scores below 500 were required to withdraw. The most impressive finding regarding 
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admission characteristics was the strong relationship between the College admission rating and 
graduation rates. As shown in Figure 1, some 98 percent of entering freshmen with admission 
ratings of 34 or higher, compared with 81 percent of those with admission ratings less than 30, 
graduated. 
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Figure 1 

Graduation Rate by Admission Rating 
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Academic Performance. T test analyses identified significant differences in academic 
performance by gender. Female annual cumulative averages, which ranged from 2.84 to 3.07, 
were consistently higher than the male averages, ranging from 2.66 to 2.96. The cumulative 
averages of U. S. citizens were somewhat, though not significantly, higher than those of 
international students. 

Analyses of variance identified no statistically significant differences in cumulative averages 
by SAT Math scores. However, significant differences were found by SAT Verbal scores. For 
example, the first year mean cumulative average for students with scores from 700 to 800 was 
3.09, compared with 2.69 for students with scores below 500 (F = 4.37, p < .001). Similarly, the 
fourth year mean cumulative average for students with scores of 650 to 690 was 3.29, compared 
with 3.04 for students with scores below 500 (F = 4.01, p < .001). 

As shown in Table 1, analyses of variance also identified statistically significant relationships 
between the admission rating and annual cumulative averages. Students with admission ratings 
of 34 to 40.5 consistently earned averages of 3. 10 or higher, while students with admission 
ratings below 30 earned averages of 2.82 or lower. 
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Table 1. Students' Mean Cumulative Averages by Admission Rating 



Admission Rating First Year Second Year Third Year Fourth Year 



34.0 to 40.5 


3.10 


3.13 


3.20 


3.28 


32.0 to 33.5 


2.78 


2.84 


2.96 


3.04 


30.0 to 31.5 


2.64 


2.73 


2.88 


2.96 


17.0 to 29.5 


2.44 


2.49 


2.67 


2.82 


F Ratio 


23.59*** 


28.47*** 


24.36*** 


20.76*** 



*** p< .001 

Significant correlations, from .44 to .47, were found between the admission rating and annual 
cumulative averages and regression results showed that the admission rating explained 2 1 
percent of the variance in final cumulative averages. These results confirm the admission rating 
as an excellent indicator of potential for academic achievement. 

Admission Characteristics, Academic Performance and Senior Survey Results. Further 
analyses were conducted to examine relationships between graduating seniors’ admission 
characteristics, academic perfonnance and senior survey responses. Admission data included 
Verbal and Math SAT scores. Academic perfonnance was based on the annual grade point 
averages and senior survey data reflected graduating seniors’ evaluation of their education and 
satisfaction with their college experience. 

Chi-Square analyses identified statistically significant relationships between the admission 
rating and satisfaction with faculty attitude (X~ = 14.09, p < .05) and re-evaluation of the college 
choice (X 2 = 20.05, p < .05). As shown in Table 2, those with the highest admission rating 
reported the highest level of satisfaction with faculty attitude and were most likely to report they 
definitely would choose the College again. 

Table 2. Admission Characteristics and Senior Survey Results 



A. Satisfaction with Faculty Attitude by Admission Rating 



Admission Rating 


Very or Generally 
Dissatisfied 


Generally 

Satisfied 


Very 

Satisfied 


Total 


N 


34.0 to 40.5 


8% 


34% 


58% 


100% 


65 


32.0 to 33.5 


4 


47 


49 


100 


53 


30.0 to 31.5 


11 


60 


29 


100 


62 


17.0 to 29.5 


12 


38 


50 


100 


42 


Total 


9 


45 


46 


100 


222 



X 2 = 14.09; p < .05 
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Table 2. Admission Characteristics and Senior Survey Results (continued) 



B. Re-Evaluation of College Choice by Admission Rating 



Admission Rating 


Probably or 
Definitely Not 


Maybe 


Probably 

Would 


Definitely 

Would 


Total 


N 


34.0 to 40.5 


24% 


12% 


38% 


26% 


100% 


66 


32.0 to 33.5 


38 


28 


25 


9 


100 


53 


30.0 to 31.5 


33 


27 


22 


18 


100 


63 


17.0 to 29.5 


26 


12 


48 


14 


100 


42 


Total 


30 


20 


32 


18 


100 


224 



y 2 = 20.45; p < .05 



Several statistically significant relationships were found between SAT scores and graduating 
seniors’ perception of the impact of their education on their abilities. As shown in Table 3, 
generally students with the lowest SAT scores reported the highest ratings. For example, 
students with SAT Verbal scores less than 500 reported a mean rating of 3.48, compared with 
3.10 among those with scores above 590, for the impact of their undergraduate education on their 
ability to think analytically and logically. Results presented in Table 3 suggest that students with 
lower ability on entrance may have a greater potential for improvement and therefore rate the 
effect of their education higher regarding the impact on their abilities and knowledge. 
Alternatively, those with higher SAT scores may report lower ratings due to higher expectations. 

Table 3. Significant Differences in Abilities Enhanced by SAT Scores 

A. SAT Verbal Scores 



Abilities 


<500 


Mean Rating 
510 to 590 


>590 


F Ratio 


Competitiveness 


3.53 


3.23 


3.19 


3.05 * 


Think Analytically and Logically 


3.48 


3.22 


3.10 


4.02 * 


Drive to Achieve 


3.43 


3.30 


3.06 


3.60 * 


Self Understanding 


3.15 


2.85 


2.71 


3.34 * 


Relate to Different People 


3.10 


2.64 


2.53 


4.38 * 
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Table 3. Significant Differences in Abilities Enhanced by SAT Scores (continued) 

B. SAT Math Scores 



Abilities 


<600 


610 to 650 


>650 


F Ratio 


Acquire New Skills and Knowledge 


3.74 


3.56 


3.47 


4.23 * 


Drive to Achieve 


3.41 


3.21 


3.00 


5.38 ** 


Think Analytically and Logically 


3.33 


3.25 


3.03 


3.33 * 


Use Quantitative Tools 


3.09 


3.25 


2.85 


4.63 * 


Formulate Creative Original Ideas 


3.08 


2.99 


2.73 


3.86 * 



Note : These mean ratings are based on the following scale: 1 'Not at AH', 2 'A Little', 3 'Moderately', and 4 'Greatly' 

* p < .05; ** p < .01 

Analyses of variance identified many significant relationships between students’ assessment 
of their undergraduate education and academic performance. For example, students who 
perceived a stronger impact on their ability to gain in-depth knowledge of a field (F=3.56, p < 
.05) and develop intellectual self-confidence (F=3.80, p < .05) earned higher fourth year 
cumulative averages. Those who reported a greater impact on their ability to use quantitative 
tools (F=3.05, p < .05) achieved higher third year cumulative averages. Analyses of variance 
also identified statistically significant relationships between students’ satisfaction with aspects of 
academic life and their annual cumulative averages. Those who were most satisfied with faculty 
attitude (F=8.33, p < .001) and academic advising (F=4.71 ,P< .01) earned the highest fourth 
year grade point average. 

Further analyses identified significant relationships between satisfaction with aspects of 
college life and academic performance. Students who reported the highest level of satisfaction 
with the following aspects of college: sense of community (F=3.1 1, p < .05), social life (F=6.61, 
p < .01) and students’ voice in policies (F=6.77, p < .01) earned the highest fourth year grade 
point averages. Those who reported the highest overall satisfaction (F=3.26, p < .05) also earned 
the highest fourth year grade point average. 

Predicting Academic Performance. Two regression analyses were performed to identify 
predictors of students’ academic performance and satisfaction with their education. In both 
regressions, demographic variables were entered first followed by admission characteristics and 
college experience variables. Table 4 presents results from the regression analysis predicting 
students’ final grade point average. 



Page 26 




Table 4 . Multiple Regression Results: Predicting Final Grade Point Average 



Predictors 


Beta 

Coefficient 


t 

Ratio 


R 2 


F Ratio 


Demographic Variables 










Gender 


-.01 


-.14 






Citizenship 


.04 


.80 


.01 


.68 



Admission Variable 

Admission Rating 


.09 


1 97*** 


.23 


20.97*** 


College Experience Variables 

First Year Fall Grade Point Average 


.74 


16.17*** 






Satisfaction with Academic Advising 


.10 


2.63** 






Satisfaction with Faculty Availability 


.11 


2 72*** 


.69 


77.69*** 



** p < .01; *** p < .001 

As shown in Table 4, among the demographic and admission variables, only the admission 
rating was a significant predictor of final grade point average. The beta coefficient of .09 
indicates that for every one unit change in the admission rating, there was a .09 unit change in 
the grade point average. As reflected in the beta coefficients, the strongest predictors of final 
grade point average were the college experience variables, including the first year fall semester 
cumulative average ( b = .74), followed by satisfaction with academic advising ( b = .10) and 
faculty availability (b = . 1 1). The R of .69 indicates that these variables explained 69 percent of 
the variance in the final grade point average. 

Predicting Overall Satisfaction. Table 5 presents results from the second regression analysis 
predicting graduating seniors’ overall satisfaction with their undergraduate education. As 
shown, citizenship was a significant predictor. The negative coefficient indicates that domestic 
students were more satisfied than international students. As denoted by the significant beta 
coefficients, students who perceived a greater impact of their education on their intellectual self- 
confidence (b=. 21) and their ability to acquire new skills and knowledge (b=. 15) were more 
likely to be satisfied. Similarly, students who expressed a higher level of satisfaction with 
faculty attitude (b=.13), the quality of business instruction (b=. 1 3) and sense of community 
(b=. 12) were also more likely to be satisfied with their undergraduate education. As indicated by 
the R 2 of .28, these variables explained 28 percent of the variance in graduating seniors’ overall 
satisfaction. 
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Table 5. Multiple Regression Results: Predicting Overall Satisfaction 



Predictors 


Beta 

Coefficient 


t 

Ratio 


R2 


F Ratio 


Demographic Variables 










Gender 


.09 


1.38 






Citizenship 


-.16 


-2.66** 


.05 


5.16** 



Perceived Impact of College 

Intellectual Self-Confidence 

Ability to Acquire New Skills and Knowledge 


.21 

.15 


2.94** 

2.09* 


.22 


16.14*** 


Satisfaction with College Experiences 










Faculty Attitude 


.13 


1.99* 






Quality of Business Instruction 


.13 


1.96* 






Sense of Community 


.12 


1.97* 


.28 


11.85*** 



* p < .05; ** p < .01; *** p < .001 



Discussion 

Results from this study bear important implications for policy and planning. The strong 
relationship found between the admission rating and graduation rate indicates that lowering the 
admission rating criterion would potentially lower the institution’s retention rate. The 
relationship found between SAT scores and retention is also important since all non-graduates 
with high SAT scores left voluntarily. Exit or follow-up interviews should be conducted to 
determine why the institution is losing these high quality students. Also notable is the fact that 
all non-graduates with SAT Math scores below 500 were required to withdraw. This finding 
suggests that an SAT Math score of 500 might be the appropriate minimum criterion for 
acceptance. 

Correlation and regression analyses validated the admission rating as an excellent indicator 
of potential for academic achievement and willingness to choose the same institution again. 
These findings identify the admission rating as a reliable measure for selecting students who are 
most likely to succeed academically and be satisfied with their college experience at this 
institution. 

Results from analyses of the relationship between SAT scores and senior survey results 
demonstrate the importance of including input characteristics in a retention/ assessment model. 
Generally students with the lowest SAT scores reported the highest rating for the impact of their 
education on various abilities, including using quantitative tools, thinking analytically and 
formulating creative ideas. These results imply that perceived impact may be affected by 
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students’ abilities. Lower ability students may have greater potential for improvement and 
therefore may report higher ratings, while high ability students may have higher expectations and 
consequently may report lower ratings. Understanding this relationship is important in analyzing 
trend data. As the quality of the student body increases, students’ assessment of their education 
may decrease. Therefore, it is necessary to control for input characteristics in analyzing outcome 
measures. 

Regression results identified student characteristics, perceived assessment and college 
experiences as significant predictors of academic performance and satisfaction with college. 
Clearly, the strongest predictor of the final grade point average was the first year fall semester 
grade point average. Other significant predictors were satisfaction with academic advising and 
faculty availability. These findings imply that the college should carefully monitor first semester 
grades and seek to increase satisfaction with advising and faculty availability to promote high 
academic performance. 

Regression results also identified perceived assessment of the college’s impact on intellectual 
self-confidence and ability to acquire new skills and knowledge and satisfaction with the quality 
of business courses, faculty attitude and sense of community as significant predictors of students’ 
overall satisfaction. These findings indicate that the College should strive to enhance these 
personal, social and academic factors to foster increased satisfaction with the college experience. 

Recommendations 

As noted previously, the organization of institutional data limited the scope of this study. 
Therefore, the final report concluded with several recommendations designed to promote the 
development of a more comprehensive retention/assessment model; enhance the efficiency of the 
process; and expand the capacity for conducting research with the College’s administrative data. 
Illustrative recommendations included the following: 

• The Admission and/or Registrar Office should maintain admission data on the 
computer system file for each entering class for at least a six year period. 

• The student administrative file should be expanded to include summary information 
regarding students’ involvement in co-curricular activities. 

• The College should develop a system for using a common ID on major 

institutional surveys and administrative student files. 

• The current model for this retention study should be expanded to create a 

comprehensive model for indirect assessment studies. 
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Conclusions 



Substantive findings include confirmation of the admission rating and SAT Verbal scores as 
valid indicators of potential for academic achievement; the significance of first semester grades 
and satisfaction with academic advising and faculty attitude to academic performance; and the 
relevance of student satisfaction with faculty attitude, the quality of business courses and sense 
of community to overall satisfaction. Finally, results showing that students with lower SAT 
scores report a greater impact of their education on their abilities emphasize the need to include 
input characteristics in a retention/assessment model in order to control for entering 
characteristics in assessing outcomes. 
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Abstract 

Institutions of higher education struggle to find ways to do effective, consistent, regular 
assessment of actual student learning, even as their faculties grow in understanding of 
assessment practices at the classroom level. One main barrier to such an institutional level of 
outcomes assessment is the lack of a model for applying available tools and practices of data 
gathering, to the otherwise individualistic practices of classroom based education. 

This work share will introduce a project currently piloted by at least nine colleges in the 
U.S. this year including Quinebaug Valley Community College, Danielson CT. The project aims 
to engage faculty and their institutions in the process of defining and consistently assessing 
learning outcomes in and across disciplines, through the use of an emerging model and tool of 
distributed data gathering. 



An Assessment Project in Context 

In the fall of 2004, faculty at Quinebaug Valley Community College (QVCC), with a 
headcount of 1,690 and an FTE of 914, began a pilot project in learning outcomes assessment 
that involves the gathering of data about actual student learning-outcomes in a range of diverse 
disciplines. Using a web-enabled, database-driven system, nineteen full-time faculty (out of 24 
who teach at the college) identified a focused selection of intended learning outcomes for the 
purposes of this data-gathering process. The faculty linked outcomes to assignments already a 
part of their courses, and developed a common set of assessment standards or rubrics. With 
these, they have begun to assess in a comprehensive way the actual learning achievements 
demonstrated by students in their respective classes. In this pilot, QVCC joins at least eight 
other colleges, each of which has begun to use the eLumen software during the 2004-2005 
academic year (Donohue-Lynch, 2005). These include Rochester Community and Technical 
College, Roch, MN; Hershberger College of Business, St. Cloud State U.; Normandale 
Community College, Bloomington, MN; North Dakota State College of Science, Wahpeton, ND; 
Anoka Ramsey Community College, Cambridge, MN; Concordia U., St. Paul, MN; Kirkwood 
Community College, Cedar Rapids, IA; Northwest Technical College, Bemidji, MN. 

At QVCC, this process has built on nearly ten years of prior efforts among the faculty to 
develop what in the words of assessment practitioners has been called a "culture of 
assessment. "(Lakos and Phipps 2002; Carter 2004) Over that prior period, faculty had been 
engaged in a range of individual and group strategies that at least to some may have seemed like 
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little more than discontinuous trendy preoccupations, faculty driven or otherwise. This apparent 
discontinuity has in fact been a particularly persistent and important symptom not just for 
Quinebaug Valley Community College but for many other institutions as well, poorly understood 
as hardly more than a sign of academia being prone to the latest professional fads. This 
challenge has been discussed in positive terms in a key work by Costa and Kallick (1995). 

The resource used in the current project at QVCC however— a data collection and 
reporting system— gives faculty individually and collectively a practical way to move from an 
ideology of consistent, regular, cross disciplinary learning outcomes assessment (Berlin and 
Donohue-Lynch 1999), to the actual practice of such assessment. In other terms, this tool 
provides a comprehensive and systematic framework within which these seemingly 
discontinuous and diverse efforts find a new coherence and practical utility for the improvement 
of teaching and learning. 



An Emerging Framework 



The software in use at the college is called "eLumen Achievements," and is one of a 
number of recently emerging solutions for such a comprehensive, systems approach to learning 
outcomes assessment (Steele and Parrinello, 2005; Tibrewal and Sobh, 2004; Pillaninayagam 
2004). 1 As such it does not do assessment itself, but instead provides a tool, (very much the way 
that a spreadsheet is now an indispensable tool in bookkeeping and accounting) through which 
faculty can both individually and collectively gather meaningful data from what they already do 
in their teaching and curriculum development. In this, it depends on and draws from already- 
occurring efforts such as classroom assessments (Angelo and Cross 1993), course-embedded 
assessments, traditional (and non-traditional) course projects, and even co-curricular activities, 
that any faculty would wish to track in a consistent and regular way. Likewise, it also calls for, 
relies on, and contributes to an effective mapping of curriculum within and across which such 
assessments are carried out (Stiehl and Lewchuk 2002a, 2002b, 2005). 

As most faculty have quickly discovered in the early stages of this project, once the 
question of learning outcomes is opened within a broader systemic framework, it challenges and 
encourages them to look in two simultaneous directions— within their own disciplines, and across 
disciplines. Each discipline area in fact now has a framework and a language within which to ask 
"what specifically should students demonstrate that they have learned through the courses we 
expect them to take?" At the same time faculty across disciplines have this as a shared 
framework as well, within which to identify and assess any subsequent achievements students 
demonstrate, wherever they occur (Schuyler 1998). 

Suddenly, potential learning outcomes look like things that could happen (or could be 
deliberately cultivated) in a variety of places rather than exclusively in one discipline or another, 
and the database in use gives a practical way to keep track of such outcomes wherever they are 
assessed. Imagine for example, students demonstrating clearly assessed writing achievements in 
a cultural anthropology class-and these being recorded in a comprehensive database (Ah! 
"Writing Across the Curriculum" !). Or imagine richly demonstrated and assessed achievements 
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related to cross-cultural understanding in a Fine Arts course; or the same for critical thinking 
skills in one or more places, and so on. 

Further, with the resulting data, curriculum can be evaluated for its integrity and 
effectiveness, and subsequent planning can be carried out to improve offerings where gaps in the 
curriculum appear, both within and across disciplines, majors, programs, and degrees. As well, 
each student eventually will likewise have her/his own individual data in the form of "learning 
outcomes transcripts," to show not only what courses they have taken or what grades they 
received in their academic career, but most importantly what actual learning outcomes they have 
achieved and to what degree of quality or depth. 

Of course, while the eLumen Achievement software is QVCC's current tool of choice for 
this project, it is the more general point that is key here: whether we choose eLumen or some 
other such software yet to be developed, it is ultimately the inherent capability within the 
software that is new and that brings assessment to a new level of coherence. This capability 
finally allows practitioners of assessment (which is just about anyone who teaches) to get beyond 
the accumulation of anecdotes and learning artifacts that demonstrate actual student learning 
achievements, to the systematic and consistent gathering of learning outcomes data that can then 
be made available in meaningful reports. 



The Pilot Project and Beyond 

Actual data-gathering in the Quinebaug Valley Community College project was off to a 
slow start after the first two semesters (fall 2004, spring 2005). While this was disappointing on 
one hand, resulting in initially sparse data, some of the reasons for this, on the other hand were 
actually quite encouraging. A number of faculty, for example, who had already engaged in the 
preliminary definition of intended outcomes, potential assessments and related rubrics, found 
that after they had gotten to the point of finally assessing actual student achievements using what 
they had initially set up, they now saw their work with a new perspective. As a result, they felt 
that they already needed to go back and redesign assessments and rubrics with more precision 
and clarity. Several faculty in fact independently began to rethink how they were approaching 
the project, and before they even input their fall 2004 data, began working to redesign and 
improve their approach for the spring semester. At least one other participant used the whole 
setup process itself as a way to engage the faculty in his area (Fine Arts) in a thorough program 
review and revision which continued into the spring 2005 semester. 

At the same time, even before any learning outcomes data had been entered into the 
database, the project could produce information that the college never had before in an easily 
accessible form: just from the preliminary setup of the eLumen system and the initial 
participating classes, the college now had an active database of intended learning outcomes, and 
a growing library of potential assignments that could be used to demonstrate and assess those 
learning achievements, mapped to current courses and programs. This in itself will be a growing 
part of the project as new courses are involved in assessment and new assignments are defined, 
toward further intended learning outcomes. 
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Meanwhile, broader, key dimensions of an holistic, systemic process of assessment are 
also well in place as part of the necessary language and practice within which a tool such as 
eLumen makes comprehensive sense. These dimensions include: 

• A respect for the professional autonomy of those who design curriculum and implement 
it in the classroom (Costa and Kallick 1995) 

• A practical respect for the individuality of students — in their diversity of learning styles, 
learning barriers and strengths, college and life-goals, and life experience (Bain 2004; 
Leamnson 2001;Schuyler 1998; Tagg 2003; Workgroup 1997) 

• The language and practice of "continuous improvement" (Freed 1997; Hatfield 1999) 

• The shift in language from what we (as teachers and administrators) do, to what students 
should actually learn (Schuyler 1998) 

• The identification and mapping of intended "learning outcomes" within and across 
curriculum (Berlin and Guan 1999; Stiehl and Lewchuk 2005) 

• The establishment of standards of assessment (rubrics) of students' work, in verifying and 
tracking actual learning achievements 

• The practice of assessment within classes and courses 

• Dialogue among faculty to encourage and support the ongoing development of 
institution-wide assessment, within and across disciplines (NEEAN)“ 



An Emerging Model of Assessment 

The ongoing challenge is to continue to develop effective, ground-level (local) practice of 
learning-outcomes assessment that at the same time relates to the wider institution within which 
it occurs. One key risk in this is the potential for constraining the professionalism and autonomy 
of those who teach (often expressed in the language of “academic freedom”), while answering 
the growing call for “accountability” in higher education (Baker 2005; Petrides 2004; Smith and 
Ruff 1998). The opposite risk is the ongoing lack of an institutionally relevant and coherent 
approach to assessment, as individual autonomy is protected and defended. A tool such as 
eLumen does not automatically protect against either of these extremes, but it does in fact 
provide a data-gathering instrument that in its very structure and function reflects the qualities of 
individual autonomy in the local practice of teaching, and systemic coherence at the institutional 
level about which we are concerned. 

The emerging model of learning-assessment represented and made more practicable by a 
tool such as eLumen is something that could not have been implemented easily before the advent 
of relational databases. Now that we have these, we are able to engage a broad range of data- 
gathering participants, few of whom ever need to understand the workings of the whole system. 
People can be expert in their own areas, for defining standards, developing effective assessments, 
creating benchmarks, and more; they can also work with other disciplines to clarify standards 
and intended outcomes. The underlying system doesn’t require that everyone understand the 
whole “big picture” of institutional assessment, curriculum mapping, or the structure of the 
database itself. At the same time, it gives people the flexibility and the potential for seeing and 
working on aspects of this “big picture,” as these emerge from the data that is eventually 
gathered. 
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The logic of such an holistic system — and its practical development in something like 
eLumen — seems sound. It is in fact a taken-for-granted model of distributed data gathering that 
is behind so many functions of everyday life today. Even within higher education itself it is a 
growing and indispensable model behind many aspects of student information management. 

And yet we lag in the extreme in any practical application of this model for the task of effective 
learning outcomes assessment. 

What is called for now are practitioners who can evaluate examples of systems such as 
eLumen both for their theoretical grounding as well as for their practical utility. Especially 
within the community college setting where the two-year focus of the institution challenges 
faculty in almost every discipline to think across typical academic boundaries, such a tool seems 
to offer a practical way to cross those boundaries while assessing and maintaining the 
institutional integrity of what we offer. The “proof of concept” will be in the data we eventually 
gather. The data will emerge if we implement and refine the right tools. The tools are beginning 
to take shape, and call for people with enough of an holistic vision of learning-outcomes 
assessment to recognize their emerging and potential value. 
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AN ANALYSIS OF GRADUATION RATES AND EXPENDITURES AT PEER 

INSTITUTIONS 



Arthur Kramer, 

Director of Institutional Research 
New Jersey City University 

Summary 

A dataset of public institutions located in medium sized cities was selected from the 
National Center for Education Statistics’ (NCES) IPEDS (Integrated Post-secondary Education 
Data System) database. The dataset consisted of the graduation rates of the 1997 first-time full- 
time cohort, the ethnic breakdown of the student population, and expenditure data for the six 
years fiscal 1998 through 2003. An analysis of the contribution to the graduation rates provided 
for by categories of expenditure data showed the greatest impact emanated from the dollars spent 
in academic support and student services. Because the data set contained a heterogeneous 
sample, consisting of doctoral degree granting and baccalaureate degree as highest degree 
institutions, a sub-sample of master’s degree institutions was selected from the data set and 
analyzed. Although the amount of variance explained in the graduation rates was attenuated by 
the selection of this sample, the model obtained statistical significance and explained over 30% 
of the graduation rates among these institutions. The data showed the greatest amount of 
variability in graduation rates was explained by the expenditures in the area of academic support; 
the minority enrollment profile of the class significantly and negatively impacted on the 
graduation rates, and for every $1 million spent in the area of academic support graduation rates 
increased by .13%; for every $1 million spent in the area of student services graduation rates 
increased by .16%. 

A comparison of the percentages of expenditures apportioned to all the categories by total 
sample, the Master’s institutions and the Master’s institutions with graduation rates above the 
median of 38% showed differences in the areas of research and public service — greater 
percentages at the total sample — academic, institutional, and instructional support — more at the 
Master’s-granting institutions. 



Methodology 

Using the IPEDS Peer analysis tool, 190 institutions were selected from the IPEDS 
database. They were selected on the basis of a “lynchpin” institution and the criteria of being 
located in a city of less than 250,000 people and under public control, as submitted on the IPEDS 
Institutional Characteristics form in fall 2004. This file was combined with a file of institutions 
previously selected by the lynchpin institution’s President’s cabinet. The resultant file of 223 
institutions was analyzed utilizing an agglomerative cluster method based on degree of 
urbanization of the institutions (i.e., located in a medium sized city of less than 250,000 people), 
Carnegie classification (classification 21: Master’s I institutions), and percent of fall 2003 
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enrollment that was African American, Hispanic, and Native American 1 . The institutions found 
to be similar to the lynchpin based on the squared Euclidean distance were: 



Auburn University-Montgomery 
California State University-Bakersfield 
Colorado State University-Pueblo 
Columbus State University 
Northeastern Illinois University 
Kean University 
Saint Peters College 
CUNY Hunter College 
Cameron University 
Texas A & M University-Corpus Christi 
The University Of Texas 
Of The Permian Basin 

In an effort to assess the extent to which operating expenses may affect one aspect of 
institutional effectiveness — graduation rates — an analysis regressing the rates onto academic 
support, institutional support, student services, research, public service, and instruction expenses 
(definitions are in the appendix), as contained in the IPEDS data base, was performed utilizing 
the entire data file (of the 223 institutions in the file, only 168 supplied complete data and were 
used in the analysis). It was found that the mean graduation rate of the 1997 first-time full-time 
cohort for all 168 institutions was M=44.3 (sd.=17.74) (table 1 — the amounts in table one for the 
expenditure means are in millions of dollars, i.e., the mean for student services was 
$67,412,100). The average size of the cohorts in the population was M=1500; sd.=1371: 
median=998; range 6809, min=6, max=6515. The expense data used were the totals from six 
years as reported in the IPEDS surveys for 1998 through 2003 (descriptive statistics for these 
variables are found in table 1). A preliminary analysis of the expense categories found they were 
all positively and significantly correlated. In order to minimize the redundancy among the 
categories in constructing a predictive model each categories was regressed onto graduation rate 
to asses their unique contributions. Iteratively subjecting the data file to a regression procedure 2 
with a different expenditure data category allowed for the elimination of three categories: 
instruction, public service, and research. Public service and research did not obtain statistically 
significant correlation coefficients with the graduation rate data and had large predictive errors 
(i.e., large “least squares residuals”). The instruction category, although statistically significant, 
also had large predictive errors and was highly correlated with the academic support category, 
which had low residual errors. When the four remaining categories were regressed onto the 
graduation rates three were found to have positive impacts and obtained statistical significance 
(table 2). The model accounted for approximately 52% of the variation in graduation rates; 



Troy State University-Montgomery 

California State University-San Bernardino 

Florida Atlantic University-Boca Raton 

Macon State College 

Indiana University-Northwest 

Rutgers University-Newark 

New Mexico State University-Main Campus 

SUNY College At Old Westbury 

Francis Marion University 

University Of Houston-Victoria 

Texas Woman's University 



A similar analysis was performed that included Asian/Pacific Islander, but the list included institutions such as UCLA, which have large 
numbers of high performing Asian students. For that reason, “minority” enrollment was limited to African American, Hispanic and Native 
American. 

2 

Stepwise methods entail each predictor variable being entered individually and tested for its impact on the dependent variable. If the predictor 
does not obtain statistical significance it is not included in the final model. The judgment of parsimony was made by saving the residuals of the 
regression models and observing the variance of the residuals and the amount of change in the R sq accompanying the inclusion of the individual 
variables. If a variable caused the variance of the residual scores to increase and did not provide for a statistically significant change in the R sq, 
the variable was not included in the final model. 
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however, as selected through the stepwise procedure, academic support expenditures accounted 
for 49% of the variation and student services and institutional support accounted for 2% and 1%, 
respectively (table 3). Table 2 also alludes to $3.9 million over six years, or approximately 
$650,000 per year, directly contributing to increasing the graduation rate at these institutions 3 (B 
coefficients in table 2). Similarly, approximately $670,000 of the monies provided for student 
services and $570,000 of the amount for institutional support increased graduation. The entire 
model obtained statistical significance F(3,164)=60.77; p=.000 (table 4). The stability of the 
estimates of the regression coefficients, i.e., an estimate of the generalizability across samples, 
shows academic support to be the most stable estimate assessed by the narrowness of the width 
of the 95% confidence interval (table 2). 



Table 1. 

Descriptive Statistics 





Mean 


Std. Deviation 


N 


Graduation rate of 
full-time students - 


44.43 


17.737 


168 


1997 cohort 
Student Services 


67.4121 


69.37674 


168 


Academic Support 


129.7332 


172.20928 


168 


Institutional Support 


113.8019 


116.48951 


168 



Table 2. 



Coefficients(a) 
















Model 




Unstandardized 

Coefficients 


Standardized 

Coefficients 


t 


Sig. 


95% Confidence 
Interval for B 






B 


Std. 

Error 


Beta 






Lower 

Bound 


Upper 

Bound 


Step 1 


(Constant) 


35.07 


1.23 




28.57 


0.00 


32.65 


37.49 




Academic Support 


0.07 


0.01 


0.70 


12.65 


0.00 


0.06 


0.08 


Step 2 


(Constant) 


33.32 


1.35 




24.72 


0.00 


30.66 


35.99 




Academic Support 


0.05 


0.01 


0.44 


4.10 


0.00 


0.02 


0.07 




Student Services 


0.08 


0.03 


0.30 


2.85 


0.00 


0.02 


0.13 


Step 3 


(Constant) 


32.38 


1.42 




22.83 


0.00 


29.58 


35.18 




Academic Support 


0.03 


0.01 


0.29 


2.28 


0.02 


0.00 


0.06 




Student Services 


0.06 


0.03 


0.25 


2.23 


0.03 


0.01 


0.12 




Institutional Support 


0.03 


0.02 


0.23 


1.99 


0.05 


0.00 


0.07 


A 


Dependent Variable: Graduation rate of full-time students - 1997 cohort 







3 (.03)( 129.7 )=3.9; 3.9/6=65. 



Page 40 






Table 3. 

Model Summar^ 



Model 


R 


R Square 


Adjusted 

R 

Square 


Std. Error of 
the 

Estimate 


Change Statistics 


R 

Square 

Change 


F Change 


df 1 


df2 


Sig. F 
Change 


1 


.701 a 


.491 


.488 


12.692 


.491 


160.139 


1 


166 


.000 


2 


.71 8 b 


.515 


.509 


12.427 


.024 


8.149 


1 


165 


.005 


3 


.726° 


.526 


.518 


12.317 


.011 


3.974 


1 


164 


.048 



a- Predictors: (Constant), Academic Support 

b- Predictors: (Constant), Academic Support, Student Services 

c. Predictors: (Constant), Academic Support, Student Services, Institutional Support 

d- Dependent Variable: Graduation rate of full-time students - 1 997 cohort 



Table 4. 

ANOVA d 



Model 


Sum of 
Squares 


df 


Mean Square 


F 


Sig. 


1 


Regression 


25796.148 


1 


25796.148 


160.139 


,000 a 




Residual 


26740.215 


166 


161.086 








Total 


52536.363 


167 








2 


Regression 


27054.620 


2 


13527.310 


87.592 


,000 b 




Residual 


25481 .743 


165 


154.435 








Total 


52536.363 


167 








3 


Regression 


27657.507 


3 


9219.169 


60.772 


,000 c 




Residual 


24878.856 


164 


151.700 








Total 


52536.363 


167 









a- Predictors: (Constant), Academic Support 

b- Predictors: (Constant), Academic Support, Student Services 

c. Predictors: (Constant), Academic Support, Student Services, Institutional Support 

d- Dependent Variable: Graduation rate of full-time students - 1997 cohort 

Adding enrollment data 

The percentages of Black/Hispanic/Native American students (“blaminhisp”) were 
combined into a single variable and entered into the regression model. The model was 
constructed hierarchically at this point, entering the ethnic variable first, as a controlling variable 
because, at least anecdotally, institutions with large minority enrollments are seen as having 
lower graduation rates, and because it was a criterion in institution selection. The expenditure 
data were entered again in a stepwise fashion. The results show the percentage of minority 
students negatively impacted the graduation rates, B=-.026; t=-4.81; p=.00 (table 5). Table 5 
also shows academic support statistically provided the greatest impact of the expenditure 
components because it was selected through the stepwise method as the next variable entered 
into the model, positively affecting graduation rates, B=.07; t=l 1.91; p=.00 (table 5, step 2). In 
this model, however, institutional support precedes student services in entry into the model. 
However, the magnitudes of these coefficients were attenuated by the addition of the enrollment 
characteristics. This may be the result of the instability of the coefficients obtained in the 
previous model (table 5). 
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The ethnicity data accounts for approximately 12% of the variation in the graduation 
rates (table 6: Rsq=.123). With the addition of the academic support data 41% more of the 
variation is explained (Rsq. Change=.406), and institutional support and student services 
expenditures contribute 2% and 1% respectively (table 6). The complete model is found to be 
statistically significant at F(4,163)=52.22; p=.000. 

Table 5 



Model 




Unstandardized 

Coefficients 




Standardized 

Coefficients 


t 


Sig. 


95% Confidence 
Interval for B 






B 


Std. 

Error 


Beta 






Lower 

Bound 


Upper 

Bound 


Step 1 


(Constant) 


50.22 


1.76 




28.54 


0.00 


46.74 


53.69 




blaminhisp 


-0.26 


0.05 


-0.35 


-4.81 


0.00 


-0.36 


-0.15 


Step 2 


(Constant) 


38.96 


1.60 




24.32 


0.00 


35.80 


42.13 




blaminhisp 


-0.14 


0.04 


-0.20 


-3.61 


0.00 


-0.22 


-0.07 




Academic Support 


0.07 


0.01 


0.65 


11.91 


0.00 


0.06 


0.08 


Step 3 


(Constant) 


37.30 


1.67 




22.30 


0.00 


34.00 


40.61 




blaminhisp 


-0.15 


0.04 


-0.20 


-3.76 


0.00 


-0.23 


-0.07 




Academic Support 


0.04 


0.01 


0.39 


3.64 


0.00 


0.02 


0.06 




Institutional 

Support 


0.05 


0.02 


0.30 


2.86 


0.00 


0.01 


0.08 


Step 4 


(Constant) 


36.25 


1.74 




20.87 


0.00 


32.82 


39.68 




blaminhisp 


-0.14 


0.04 


-0.19 


-3.62 


0.00 


-0.22 


-0.06 




Academic Support 


0.03 


0.01 


0.26 


2.09 


0.04 


0.00 


0.05 




Institutional 

Support 


0.04 


0.02 


0.24 


2.22 


0.03 


0.00 


0.07 




Student Services 


0.05 


0.03 


0.21 


2.02 


0.05 


0.00 


0.11 


a 


Dependent Variable: Graduation rate of full-time students - 1997 cohort 









Table 6 

Model Summary(e) 



Model 


R 


R 

Square 


Adjusted 
R Square 


Std. Error 
of the 
Estimate 


Change Statistics 


R Square 
Change 


F Change 


dfl 


df2 


Sig. F 
Change 


Step 1 


.350(a) 


.123 


.117 


16.664 


.123 


23.181 


1 


166 


.000 


Step 2 


.727(b) 


.528 


.523 


12.255 


.406 


141.952 


1 


165 


.000 


Step 3 


.742(c) 


.551 


.543 


11.997 


.022 


8.178 


1 


164 


.005 


Step 4 


.749(d) 


.562 


.551 


1 1 .886 


.011 


4.067 


1 


163 


.045 



a Predictors: (Constant), blaminhisp 

b Predictors: (Constant), blaminhisp, Academic Support 

c Predictors: (Constant), blaminhisp, Academic Support, Institutional Support 

d Predictors: (Constant), blaminhisp, Academic Support, Institutional Support, Student Services 

e Dependent Variable: Graduation rate of full-time students - 1997 cohort 
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Discussion 



The series of regression analyses using the original six categories of expenditures data 
allowed for the exclusion of three categories: public service, research, and instruction. The 
iterative process with various combinations of the variables showed all were correlated among 
themselves (i.e., high collinearity) necessitating excluding and/or combining some in order to get 
better estimates of the regression coefficients. These three categories did not provide sufficient 
statistical power by themselves for inclusion 4 . The precision of the estimates of explanatory 
power was maximized while variability of the estimates of predicted scores were simultaneously 
minimized by excluding these categories. Once the satisfactory subset of variables was settled 
upon, the ethnicity data were entered as a controlling factor. The final model, which included 
the ethnicity data, explained approximately 50% of the variation among the institutions’ 
graduation rates, but the stability of the estimates of the regression coefficients is an issue 
because there is still collinearity among the expenditure variables. However, repeated analyses 
found the academic support data continuously provided the greatest impact on the graduation 
rates. Unfortunately, the IPEDS data contain no other student characteristics for the 1997 cohort 
that could have been included, such as preparedness, level of parental education, or 
programmatic information from the institutions that could possibly increase the explained 
variability over the 50%. With other available data, the expenditures could either be combined 
into one category or one or more components eliminated, to remove the remaining collinearity 
and better assess the factors impacting graduation rates. 

Analysis of a sub-group 

In reviewing the list of institutions it became apparent it was a heterogeneous group. 
There were doctoral degree-granting universities and liberal arts colleges included in the file. A 
subgroup from the file was selected consisting of only those institutions with Carnegie 
classification of 21 — Master’s degree granting. Seventy-six institutions provide sufficient data 
to perform the analysis. The mean graduation rate was M=38.6%, (stddev =13.8) (table 7). An 
interesting finding is that less of the variability among the graduation rates is explained by the 
expenditure data in these Masters’ institutions, and the institutional support variable did not 
obtain statistical significance and was omitted from the model by the stepwise regression 
algorithm, but the impact of the academic support and student services expenditures was greater 
at these institutions. In the larger group the graduation rate increased by .03% for every 
$lmillion spent in academic and institutional support and .06% for every $1 million spent on 
student services. In the sub-group, the graduation rate increased by .13% for every $1 million 
spent on academic support and .16% for every $1 million spent on student services (academic 
support and student services data positively and significantly affected the rates B=.13, 
t=2.64,p=.01; B=.16, t=2.29, p=.03, respectively (table 10). Ethnicity negatively impacted 
graduation rates: B=-.ll, t=-2.37, p=.02 (table 10). About 37% of graduation rate variance was 
explained using the current, sub-group, dataset, where as in the previous, larger group, 
approximately 50% was explained by the data (table 8). The sub-group model obtained 
statistical significance F (3,72)=13.88;p=.000 (table 9). The above suggests that $6.7 million of 



Principal components analysis was employed to construct unique categories based on the six variables. It resulted in one large component on 
which all of the variables loaded and three smaller ones. There are correlations among the factors and, rather than employing a correlational 
oblique factor rotation it was decided to eliminate variables that did not provide significant explanatory power or minimize residual variability. 
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the $51.6 million (13%) of the monies allocated to academic support over the six-year period 
contributed to the graduation rate of 38.6%, and similarly, $6.3 million of the $39.3 million 
(16%) appropriated to student services contributed to the rate. The most influential category in 
explaining the graduation rates is the academic support category uniquely explaining 23% of the 
variability, Rsq change=.23 (table 8). 



Table 7. 

Descriptive Statistics 





Mean 


Std. Deviation 


N 


Graduation rate of 
full-time students - 


38.66 


13.815 


76 


1997 cohort 
blaminhisp 


29.2618 


27.45706 


76 


Student Services 


39.3100 


23.57301 


76 


Academic Support 


51.5710 


34.44296 


76 


Institutional Support 


64.7015 


35.50211 


76 



Table 8 

Model Summary(d) 



Model 


R 


R 

Square 


Adjusted 

R 

Square 


Std. 
Error 
of the 
Estimate 


Change 

Statistics 












R 

Square 

Change 


F 

Change 


dfl 


df2 


Sig. F Change 


Step 1 


0.31 


0.10 


0.08 


13.23 


0.10 


7.79 


1.00 


74.00 


0.01 




Step 2 


0.57 


0.32 


0.30 


11.54 


0.23 


24.19 


1.00 


73.00 


0.00 




Step 3 


0.61 


0.37 


0.34 


11.22 


0.05 


5.23 


1.00 


72.00 


0.03 




a 


Predictors: (Constant), blaminhisp 














b 


Predictors: (Constant), blaminhisp. Academic Support 










c 


Predictors: (Constant), blaminhisp. Academic Support, Student Services 






d 


Dependent Variable: Graduation rate of full-time students - 1997 cohort 
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Table 9 
ANOVA(d) 



Model 




Sum of 
Squares 


df 


Mean Square 


F 


Sig. 


1 


Regression 


1362.908 


1 


1362.908 


7.787 


.007(a) 




Residual 


12951.046 


74 


175.014 








Total 


14313.954 


75 








2 


Regression 


4585.963 


2 


2292.981 


17.207 


.000(b) 




Residual 


9727.992 


73 


133.260 








Total 


14313.954 


75 








3 


Regression 


5244.581 


3 


1748.194 


13.879 


.000(c) 




Residual 


9069.373 


72 


125.964 








Total 


14313.954 


75 









a Predictors: (Constant), blaminhisp 
b Predictors: (Constant), blaminhisp, Academic Support 
c Predictors: (Constant), blaminhisp, Academic Support, Student Services 
d Dependent Variable: Graduation rate of full-time students - 1997 cohort 
Variable: Graduation rate of .Full-time students - 1997 cohort 



Table 10. 



Coefficients(a) 


Model 




Unstandardized 

Coefficients 




Standardized 

Coefficients 


t 


Sig. 


95% 

Confidence Interval 
for B 






B 


Std. 

Error 


Beta 






Lower 

Bound 


Upper 

Bound 


1 


(Constant) 


43.21 


2.23 




19.41 


0.00 


38.77 


47.64 




blaminhisp 


-0.16 


0.06 


-0.31 


-2.79 


0.01 


-0.27 


-0.04 


2 


(Constant) 


32.80 


2.87 




11.42 


0.00 


27.07 


38.52 




blaminhisp 


-0.14 


0.05 


-0.27 


-2.79 


0.01 


-0.23 


-0.04 




Academic 

Support 


0.19 


0.04 


0.48 


4.92 


0.00 


0.11 


0.27 


3 


(Constant) 


29.22 


3.20 




9.12 


0.00 


22.83 


35.60 




blaminhisp 


-0.11 


0.05 


-0.23 


-2.37 


0.02 


-0.21 


-0.02 




Academic 

Support 


0.13 


0.05 


0.31 


2.64 


0.01 


0.03 


0.22 




Student Services 


0.16 


0.07 


0.28 


2.29 


0.03 


0.02 


0.30 


a 


Dependent Variable: Graduation rate of full-time students - 1 997 cohort 
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The following list of institutions are those Master’s granting institutions whose 
graduation rate for the 1997 cohort exceeded the mean of 39%: 



California State University-Bakersfield 
California State University-San Bernardino 
Citadel Military College Of South Carolina 
CUNY Queens College 
Florida Agricultural And Mechanical 
University 

Minnesota State University-Moorhead 
North Carolina A &T State University 
Rhode Island College 
Rutgers University-Camden 
Saint Peters College 
Texas State University-San Marcos 
University Of Tennessee-Chattanooga 
University Of Minnesota-Duluth 
University Of Northern Iowa 
University Of Wisconsin-La Crosse 
Western Washington University 
William Paterson University of New Jersey 
Worcester State College 

Discussion 



California State University-Chico 
California State University-Stanislaus 
College Of Charleston 
Fitchburg State College 
Kean University 

Montclair State University 
North Carolina Central University 
Rowan University 
Saint Cloud State University 
Southwest Missouri State University 
College Of New Jersey 
University Of Michigan-Dearbom 
University Of North Carolina-Wilmington 
University Of Wisconsin-Eau Claire 
University Of Wisconsin-Oshkosh 
Westfield State College 
Winthrop University 



The smaller sample demonstrates financial and enrollment profiles impact the sub-group 
of Master’s-granting institutions differently than they impacted the larger group of all public 
institutions located in medium sized cities. One finding repeatedly surfaced in both the larger 
“population” and the sub-group sample — the monies spent supporting the academic programs 
provided the greatest impact on the graduation rates. Monies spent supporting student services 
were also found to positively impact graduation rates, but to a lesser extent, and the ethnic 
breakdown of the institutions’ students also significantly impacted the rates, but negatively. 
Another difference between the two samples is that the institutional support expenditures did not 
significantly impact the graduation rates at the Master’s-granting institutions, as they did the 
larger sample. 

Percentages of expenditures by category 

Percentages of expenditures by category were examined over the six-year period by all of 
the institutions; the Carnegie 21 (Master’s-granting) institutions; and, the Carnegie institutions 
whose graduation rates were above the median of 38%. They show that within the sample of all 
institutions a greater percentage of the expenditures went to research than at the Carnegie 21 
institutions, and less went to academic and institutional support. At the Carnegie 21 institutions, 
more money was spent in the areas of instruction and student services than in the larger sample. 
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The masters’ institutions also spent less in the areas of public service, and slightly more in the 
area of academic support. 

Comparing the two groups of Masters’ institutions, it is seen that the group that graduated 
the higher numbers of students spent more on instruction, academic support, institutional support 
and less on public service, research and student services. 

Conclusion 

Data gathered from the IPEDS Peer Analysis tool allowed for an investigation of how 
expenditures in the financial categories captured by that annual survey impact graduation rates — 
a measure selected by Federal and State legislatures to gauge institutional effectiveness. It was 
found that there was a high degree of correlation among the expenditure variables and, by means 
of successive analytic procedures, several could be removed. Of the categories that remained 
academic support and student service expenditures repeatedly emerged as providing positive and 
significant effects on the graduation rates, and with the greatest explanatory power. Included in 
the definition of academic support are references to educational materials, activities and support 
services for academic functions, media, administration, separately budgeted academic personnel 
development, and course and curriculum development. Under student services are expenses for 
admissions, registrar, “...and activities whose primary purpose is to contribute to students’ 
emotional and physical well-being and to their intellectual, cultural, and social development 
outside the context of the formal instructional program.’’ What is more, in regard to the Master’s- 
granting institutions, the degree of impact from academic support and student service was greater 
than that provided to the larger population. Although the results provide a high degree of 
explanatory information there is still much missing of what impacts graduation rates, especially 
in regard to the Carnegie 21 institutions, where almost 70% of the variance is not explained by 
the model. The institutions in the analysis conformed to a profile of “urbanicity’’, minority 
enrollment, and public control. Analysis of private institutions with different environmental 
conditions and student ethnic profile may obtain different results. 

The comparison of expenditures by institution type in figures 1 to 3 shows that, within all 
the separate analyses, the area of instruction receives the “greatest piece of the pie.” The second 
largest piece is research when all institutions in the data set are assessed, but when just masters’ 
institutions are observed institutional support receives the second largest number of dollars. 
Analysis of a qualitative nature may shed light on programs existent in all master’s-granting 
institutions, master’s-granting institutions with exemplary graduation rates, and/or institutions in 
large urban settings. Graduation rates are not only impacted by the categories in which 
institutions spend; student characteristics are more complex than the ethnic/racial components 
addressed here. Motivation and involvement in extra curricular activities and academics also 
impact student success. 
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NSSE’S BENCHMARKS - ONE SIZE FITS ALL? 



Nava Lerer and Kathryn Talley 
Office of Research, Assessment and Planning 
Adelphi University 

Administrators of the National Survey of Student Engagement (NSSE) argue that the 
results from the five benchmarks, constructed of about forty out of the survey’s eighty items, 
“produce a set of national benchmarks of good educational practice that participating schools are 
using to estimate the efficacy of their improvement efforts” (Kuh, 2001b). The objective of this 
paper is to show that any assessment of educational quality cannot be separated from students’ 
expectations and goals. To the degree that traditional and nontraditional students differ on these 
dimensions, benchmarking must carefully avoid bias toward one “type” of student over the other. 
Traditional college students (those starting as freshmen and attending college immediately after 
high school) expect a broad range of non-academic options as part of their educational 
experience, such as extracurricular activities, community involvement, and interaction with 
faculty and peers outside the classroom. Nontraditional students’ educational goals are more 
narrowly focused. We argue that three of the five NSSE benchmarks mix items that primarily 
reflect the expectations of traditional college students with more “universal” educational 
experiences that focus on academics, classroom activities and institutional support, and therefore 
are not appropriate for assessing the quality of education for all college students. The paper also 
argues for the restructuring of these problematic benchmarks to accurately reflect educational 
practices common to all types of students instead of using the current benchmarks, which 
penalize institutions with large nontraditional student populations. 

Brief Literature Review 

Developed in part as an alternative to reputation- and resource-based ranking, the NSSE 
was designed to assess the extent to which students are engaged in educationally purposeful 
activities that contribute to their learning and success during college (Kuh, 2001a, 2001b; Pike, 
2003). Based on accumulated research, starting with the “Seven Principles for Good Practice in 
Undergraduate Education” (Chickering & Gamson, 1987), NSSE administrators argue that there 
is a correlation between students’ engagement and the quality of education they receive. “As a 
survey NSSE annually assesses the extent to which students at hundreds of four-year colleges 
and universities are participating in educational practices that are strongly associated with high 
levels of learning and personal development” (Kuh, 2001a). In addition to including activities 
that are traditionally associated with learning, such as reading and writing, preparing for class, 
and interacting with instructors, “the engagement concept also encompasses some other key 
activities that more recently have come to the fore as being important, such as collaborating with 
peers on projects, problem solving tasks, and community service” (Kuh, Gonyea, & Palmer, 
2001 ). 



Using a questionnaire that consists of over eighty items, NSSE assesses student 
engagement in activities its researchers contend contribute to learning and success during 
college. In order to make the survey results more accessible and manageable, five benchmarks 
were created: “To facilitate the conversation about student engagement, learning, and 
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institutional improvement, we grouped key questions from the survey into five clusters or 
benchmarks of effective educational practices” (Kuh, 2003). The five NSSE benchmarks are: 1) 
level of academic challenge, 2) active and collaborative learning, 3) student-faculty interaction, 

4) enriching educational experiences, and 5) supportive campus environment. Overall, the 
benchmarks “are intended to help steer the national conversation about collegiate quality away 
from resources and reputational rankings toward what matters more to student learning — good 
educational practice” (NSSE 2000, p.l). Survey administrators claim that NSSE’s results, 
including the benchmarks, can be used to compare the quality of education at different 
institutions: “Those institutions that more fully engage their students in the variety of activities 
that contribute to valued outcomes of college can claim to be of higher quality in comparison 
with similar types of colleges and universities” (Kuh, 2001b). “The NSSE benchmarks are a 
window into student and institutional performance at the national, sector, and institutional levels” 
(Kuh, 2003). 

Several analyses (conducted by NSSE researchers among others) show, however, that 
nontraditional students (older, commuters, transfers) respond differently on many of the NSSE 
questions - especially those items inquiring about activities outside the classroom - and 
consequently have lower scores on several of the NSSE benchmarks. For example, students 
enrolled full-time have consistently higher scores on NSSE benchmarks, probably because full- 
time students have more opportunities to become engaged in educationally purposeful activities 
(Pike, 2003). Similar results are shown for students who entered the institution as transfers. 
“Overall, transfer students are less engaged in effective educational activities than their non- 
transfer peers. Transfer students tend to be older and have more external responsibilities such as 
working for pay off-campus and caring for dependents. Transfer students believe their 
coursework provides more emphasis on cultivating higher-order thinking abilities than their 
peers, yet they interact with faculty members and engage in enriching educational programs at 
levels lower than their counterparts” (NSSE 2004 overview, p.9). Commuter students were also 
found to be less “engaged” overall, although this was not true in the classroom (Kuh, Gonyea & 
Palmer, 2001; Chickering, 1974; Pascarella & Terenzini, 1991). Finally, older students were 
found to be less engaged in activities outside the classroom: “Younger traditional age students 
(18-24 years) report spending slightly more time in educationally productive activities and 
perceive their campus environment as more supportive than older students. However, older 
students did not differ much from their younger counterparts in educational and personal growth 
and in their perceptions of course emphasis on higher-order mental activities” (NSSE 2004 
overview, p.9). Other research has found that older seniors (those over age 25) had similar 
responses to younger seniors on most NSSE items an institution can “control” - classroom 
activities, relationships with faculty and administrators, and institutional support - but had 
different responses on items that were related to student lifestyle. These older seniors mainly 
showed less engagement in activities and less interaction with other students and faculty outside 
the classroom (Hicks & Lerer, 2002). 

The educational goals of nontraditional students cannot and should not be ignored by 
NSSE researchers as though this group is inconsequential. Quite the contrary: the proportion of 
traditional students is steadily declining on college campuses. “According to NCES, just over 
40 percent of postsecondary students today attend part time, compared with less than one-third a 
generation ago. Similarly, the proportion of students 25 and older has jumped from just over 
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one-quarter in 1970 to nearly 40 percent today. Moreover, nearly 40 percent of students now 
attend more than one institution in their college career. All in all, we live in nontraditional times 
when obtaining a ‘four year’ degree in four years is the exception rather than the rule” ( The 
Chronicle of Higher Education, June 27, 2005). Nontraditional students, who are usually older, 
live and work off-campus and have families and responsibilities not related to their experiences 
as students, do not seek the same outcomes from their education as traditional college students. 
They tend to focus on academics, and do not have the time (or perhaps the inclination) to 
participate in off-campus activities or interactions outside the classroom. 

Accordingly, NSSE should not ignore the preferences and experiences of this distinct 
group of students by expecting the same behaviors and practices from cdl college students. 
Educational practices that are applicable to all students, such as classroom-related activities and 
institutional support, should be distinguished from activities that clearly reflect preferences and 
behaviors of traditional college students, such as study abroad, participation in extracurricular 
activities, and interactions with faculty and students outside the classroom. Even Chickering & 
Gamson, whose seven principles of good educational practice serve as the basis for NSSE’s 
conceptual framework, argued against a one-size-fits-all scheme: “The ways different 
institutions implement good practice depend very much on their students and their 
circumstances” (1991). 

Combining the educational needs and preferences of traditional and nontraditional 
students under a few all-encompassing (and arguably biased) benchmarks puts institutions with 
large nontraditional student populations at a comparative disadvantage. While NSSE researchers 
clearly imply by the way the benchmarks are constructed that out-of-classroom activities are an 
essential part of students’ educational experiences, they have not proven (or even argued) that 
nontraditional students, who tend to engage in these activities less than traditional students, 
receive an inferior education. In fact, NSSE’s own researchers show that nontraditional students 
are more satisfied with their overall educational experiences (NSSE 2004 overview). 

Brief Summary of the Methodology 

To test the argument of the paper, we created distinct traditional and nontraditional 
groups using students’ age and whether they had started at Adelphi as freshmen or transfers. We 
also chose to focus on seniors, a group that includes a larger proportion of nontraditional 
students. Traditional students were defined as seniors younger than 25 who had started at 
Adelphi as freshmen. Nontraditional students were defined as seniors older than 30 (since the 
average age of Adelphi senior respondents was 30.5) who started as transfers. Seniors who did 
not fit this “pure” traditional/nontraditional typology were excluded from the analysis. Other 
independent variables included in the regression equations were on-campus residence, gender, 
ethnicity and parents’ education. 

We expected that there would be no significant difference between the two groups in 
analyzing the two benchmarks that include items applicable to all students: 1) The “level of 
academic challenge” benchmark, which focuses on classroom and academic activities; and 2) the 
“supportive campus environment” benchmark, which is applicable to the successful educational 
endeavors of all students. In contrast, we hypothesized that a large number of items in the 
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remaining benchmarks are primarily applicable to traditional students. 1) In the “student-faculty 
interaction” benchmark, a smaller proportion of older students are likely to engage in three of the 
five activities specified: fewer will need to “talk about career plans with faculty members,” or 
will have time for interacting with faculty members outside class to “discuss ideas from readings 
or classes,” or “work on activities other than coursework (committees, orientation, student-life 
activities, etc.).” They also will probably spend less time on the fourth item, “discussed grades 
or assignments with an instructor,” since this activity tends not to occur in the classroom, 
although they might speak with instructors immediately after class or e-mail them. 2) In the 
“active and collaborative learning” benchmark, three of seven items refer to activities outside the 
classroom: “worked with classmates outside class to prepare class assignments,” “tutored or 
taught other students,” and “participated in community-based project as part of a regular course.” 
3) In the “enriching educational experiences” benchmark, four of twelve variables inquire about 
out-of-classroom, non-academic activities: participating in “co-curricular activities,” “practicum, 
internship, field experience, co-op experience or clinical assignment,” “community service or 
volunteer work,” and “study abroad.” A fifth item - “learning communities” - is problematic in 
that it is usually a part of students’ first-year experience and not of students who started as 
transfers (although transfer students might think that taking classes with the same students in 
their majors is “other formal program where groups of students take two or more classes 
together”). This benchmark is also conceptually questionable since it includes seemingly 
unrelated items covering at least four areas: participation in activities outside the classroom, 
participation in activities inside the classroom, campus diversity, and technology use. 

Brief Summary of the Data Sources 

Data are drawn from two sources: NSSE’s data and Adelphi’s data. Adelphi has 
participated in NSSE each year since its introduction in spring 2000. Adelphi has five years of 
NSSE results (since NSSE created its benchmarks in 2001) at its disposal. The final file 
included 432 seniors: 192 traditional and 240 nontraditional (Table 1). 



Table 1: Respondents’ distribution within the groups 





Age 




Less than 25 


25-30 


Over 30 


Freshmen 


192 1 


10 


32 


Transfers 


165 


81 


240“ 



1 = Traditional group 



11 = Nontraditional group 

Note: On-campus residence was not used to create the traditional/nontraditional groups because the vast 
majority of all seniors live off-campus. The off-campus (commuter) specifics are: traditional group, 144 
commuters; nontraditional group, 240 commuters. 
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While NSSE created aggregate benchmarks to derive scores for an entire 
institution, we applied the same process to create benchmark scores for individual students 
- using the SPSS syntax NSSE provides on its website. These constructed benchmarks 
were the dependent variables in the analyses. Variables used as controls were those found 
in the literature to affect students’ educational experiences: entrance status, on-campus 
residents versus commuters, gender, ethnicity, and parents’ education. 

Results 

When the demographic characteristics of the traditional and nontraditional groups are 
examined, the results show a similar proportion of women. The nontraditional group includes a 
slightly higher proportion of minority students (non-white) and a considerably higher proportion 
of respondents whose parents did not graduate from college (Table 2). 



Table 2: Demographic characteristics 





Traditional 


Nontraditional 


% Women 


77.1% 


80.4% 


% Minority 


15.1% 


23.8% 


% At least one parent graduated 
from college 


60.1% 


39.5% 


% Commuters 


75.0% 


100.0% 


Average age 


21.8 


42.0 



As hypothesized, when the mean responses of the two groups to the benchmarks are 
examined, the results show similar responses to the “level of academic challenge” and 
“supportive campus environment” benchmarks, slightly larger differences to the “active and 
collaborative learning benchmark,” and markedly larger differences to the “student-faculty 
interaction” and “enriching educational experiences” benchmarks (Table 3). 



Table 3: Average scores of the benchmarks 





Traditional 


Nontraditional 


Level of academic challenge 


54.2 


53.4 


Active and collaborative learning 


49.3 


44.6 


Student-faculty interaction 


50.3 


39.3 


Enriching educational experiences 
(2004, 2005 data only) 


41.8 


30.2 


Supportive campus environment 


56.9 


59.1 



As a first step, we conducted OLS regression analyses using the five NSSE benchmarks 
as dependent variables and the traditional/nontraditional groups (nontraditional=l, traditional=0), 
gender (men=l, women=0), minority (non-white=l, white=0) and parents’ college education 
(college graduates=l, none=0) as independent variables (Table 4). 
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As hypothesized, the traditional/nontraditional variable was not a significant predictor for 
the “level of academic challenge” benchmark (p=.60) or the “supportive campus environment” 
benchmark (p=.22). Additionally, with the exception of gender in the equation where the 
“supportive campus environment” benchmark is the dependent variable, none of the other 
independent variables were significantly related to these two benchmarks. It should be noted, 
however, that virtually no variance in the “level of academic challenge” benchmark is explained 
by the demographic variables included (R =.004). Also, while the traditional/nontraditional 
variable explains 6 percent of the variance in the “supportive campus environment” benchmark, 
neither this variable nor any of the other independent variables is significantly related to this 
benchmark. 

As expected, the traditional/nontraditional variable was significant in the other 
benchmarks. Seniors in the nontraditional group had significantly lower scores on the “active 
and collaborative learning” benchmark (p=.004) and the “student-faculty interaction” benchmark 
(pc. 001). When the other independent variables are included, the traditional/nontraditional 
variable was still highly significant and, with the exception of gender in the equation where the 
“student-faculty interaction” benchmark is the dependent variable, none of the other independent 
variables is significant. The proportion of variance explained by the traditional/nontraditional 
variable for these two benchmarks is, however, very small (less than 2%). 



Table 4: OLS Regressions of NSSE’s Benchmarks (B & significance) 





Level of 
academic 
challenge 


Active and 
collaborative 
learning 


Student- 

faculty 

interaction 1 


Enriching 

educational 

experiences 11 


Supportive 

campus 

environment 


With Traditional/Nontraditional variable only 










Constant 


54.209 

(.000) 


49.071 

(.000) 


50.390 

(.000) 


41.693 

(.000) 


56.803 

(.000) 


Traditional/Nontraditional 


-.463 

(.604) 


-4.514 

(.004) 


-10.917 

(.000) 


-11.974 

(.000) 


2.244 

(.221) 


Adjusted R 2 


-.002 


.017 


.001 


.104 


.060 


With all other independent varial 
Constant 


ties 

52.765 

(.000) 


49.780 

(.000) 


48.915 

(.000) 


41.668 

(.000) 


54.707 

(.000) 



Traditional/Nontraditional 


-.477 

(.754) 


-4.796 

(.003) 


-10.131 

(.000) 


-11.156 

(.000) 


2.297 

(.223) 


Gender 


1.217 


-.576 


5.717 


5.429 


5.019 


(.501) 


(.763) 


(.024) 


(.111) 


(.025) 


Minority 


.802 


.332 


-2.830 


-5.869 


2.314 


(.655) 


(.862) 


(.260) 


(.129) 


(.298) 


Parents’ college 


1.751 

(.246) 


-1.040 

(.515) 


-.955 

(.650) 


-.984 

(.734) 


1.015 

(.587) 


Number of respondents 


418 


420 


420 


132 


417 


Adjusted R 2 


-.004 


.012 


.010 


.132 


.061 



I Excludes “working on research project with faculty member outside the classroom” 

II Includes only the 2004 and 2005 respondents 
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The benchmark most strongly related to the traditional/nontraditional variable is 
“enriching educational experiences” - nontraditional seniors have significantly lower scores on 
this benchmark than traditional seniors (pc.OOl). Moreover, belonging to one of these two 
groups by itself explained 10 percent of the variance of this benchmark, with an adjusted 
R =.098 (compared with an adjusted R“ of .01 to .03 in the other benchmarks). These results 
reinforced our earlier argument that in addition to being inconsistent conceptually, this 
benchmark also includes a large number of items that depict outside-of-classroom activities, 
which are not applicable to nontraditional students. 

We further analyze the benchmarks in which the differences between traditional and 
nontraditional seniors are significant, by indicating our opinion of the applicability of each of the 
items in the benchmarks to all students and showing the correlations between the items and the 
two groups. We hypothesize that the groups will not be significantly different for the items we 
consider as universally applicable to “good educational practices,” but significantly different for 
items that we consider as applicable solely for traditional college students (Table 5). 

Almost all items that we argued are only applicable to traditional students (less than 25 
years old who started as freshmen) compared with nontraditional students (over 30 years old who 
started as transfers) were significantly correlated with the traditional/nontraditional indicator. 
About half of the items we argued are applicable to all students, however, were significant as 
well. In the “active and collaborative learning” benchmark, nontraditional students asked 
significantly more questions and participated in class discussions (p=.008), but made fewer class 
presentations (p=.005) - two items we hypothesized are equally applicable to both groups. 

While older students probably feel less intimidated by professors and fellow students and are 
therefore more inclined to express their opinion, it is not clear why they would make fewer class 
presentations. 

Similar results are shown for the “student-faculty interaction” benchmark - the groups 
significantly differed on all the items we hypnotized they would, but were also significantly 
different on some of items we thought were equally applicable to both groups. Nontraditional 
students discussed grades or assignments significantly less than traditional students (p=.01) - an 
item we were not sure about. We would also like to add that the questionnaire item we consider 
the most important indicator for student-faculty interaction, quality of relationships with faculty 
members, should have been a part of this benchmark. 

Finally, in the “enriching educational experiences” benchmark, the two groups 
significantly differed on almost all items we thought were not equally applicable to all students. 
The only exception was the “study abroad” item; there was no significant difference between the 
groups, probably because very few traditional Adelphi students participate in this experience. In 
addition, we were not sure whether nontraditional students have the same opportunity to interact 
with diverse groups of students as do traditional students, because they spend less time on 
campus outside the classroom; the traditional and nontraditional groups responded in 
significantly different ways to two items that focused on conversations with other students. We 
were also not sure whether respondents would understand what is meant by “learning 
communities,” since many transfers might have thought that this concept applied to taking 
classes in their majors with the same students (although this is not the item’s intent) - and the 
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groups were indeed not significantly different in their response. In addition, the two groups were 
significantly different on taking foreign language courses - an item we were not sure about - 
probably because Adelphi does not require that all students take foreign languages, and students 
who started as transfers might have been either less inclined to do so or had taken these courses 
at their prior institutions. 

Table 5: Applicability of items in the three NSSE benchmarks and their correlation with 

the traditional/nontraditional indicator 



Benchmarks 


Applicability 
to all students 


Correlation 


Active and collaborative learning 






Asked questions in class or contributed to class discussions 


yes 


.127** 


Made a class presentation 


yes 


-.135** 


Worked with other students on projects during class 


yes 


-.015 


Worked with classmates outside of class to prepare class assignments 


no 


-.104* 


Tutored or taught other students 


no 


- 29 1 *** 


Participated in a community-based project as part of a regular course 


no 


-.122* 


Discussed ideas from your reading or classes with others outside of class 


yes 


-.015 


Student-faculty interaction 






Discussed grades or assignments with an instructor 


not sure 


-.157*** 


Talked about career plans with a faculty member or advisor 


no 


i 

o 

00 

* 


Discussed ideas from reading/classes with faculty members outside of class 


no 


-.267*** 


Worked with faculty members on activities other than coursework 


no 


-.294*** 


Received prompt feedback from faculty on your academic performance 


yes 


-.062 


Enriching educational experiences 






Serious conversations with students with different religious beliefs, 
political opinions, or values 


not sure 


-.256*** 


Serious conversations with students of a different race or ethnicity 


not sure 


-.222** 


An institutional climate that encourages contact among students from 
different demographic backgrounds 


yes 


-.077 


Using electronic technology to discuss, complete an assignment 


yes 


-.024 


Participating in internships, field, co-op experiences 1 


no 


-.238** 


Participating in community service or volunteer work 1 


no 


- 219*** 


Participating in foreign language coursework 1 


not sure 


-.266*** 


Participating in study abroad 1 


no 


-.139 


Participating in independent study or self-designed major 1 


yes 


-.193* 


Participating in culminating senior experience 1 


yes 


-.047 


Participating in co-curricular activities 


no 


-.257*** 


Participated in learning community/some other formal program 1 


not sure 


.010 



Significance level: * <.05; ** <.01; ***<.001 
1 l=done; 0=plan to do, do not plan to do, have not decided 
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Conclusions and Implications for Future Research 

While NSSE items are important indicators of experiences that certain types of students 
might be seeking and expect from their college, the benchmarks are supposed to provide an 
overall picture of colleges’ educational practices. The fact that students’ needs and expectations 
differ is to be expected - traditional students usually prefer educational experiences that provide 
activities outside the classroom in addition to their classroom education, while nontraditional 
students are more focused on an education that provides the academic experiences and support 
services they need, apart from extracurricular offerings and activities outside the classroom. The 
current NSSE benchmarks are biased, however, toward traditional college students. They mix 
items that clearly reflect the particular experiences of traditional students with experiences 
reflecting the quality of education that all students need and deserve. 

The results of the regression analyses support our argument. By including items in the 
benchmarks that are skewed toward the goals and expectations of traditional students, NSSE 
penalizes institutions with a high concentration of nontraditional students (especially older 
students who started as transfers). The quality of education in institutions with a large proportion 
of nontraditional students will inevitably look worse when compared on the benchmarks with 
institutions serving more traditional students. 

More specifically, we believe that a benchmark claiming to capture “student-faculty 
interaction” cannot focus on activities outside the classroom. Such a benchmark must also 
include the only item that directly asks students about their interaction with faculty - students’ 
rating of the quality of their relationships with faculty members (this item is included in the 
“supportive campus environment” benchmark instead). 

The “enriching educational experiences” benchmark is also problematic; in addition to 
the fact that this benchmark includes a large number of items that we believe are only applicable 
to traditional students, we find the benchmark itself to be conceptually muddled. While an 
enriching educational environment is very important to all students, NSSE might want to 
consider developing two benchmarks. One should cover areas applicable to all students while the 
other should cover activities that are almost exclusively applicable to traditional students, such as 
study abroad, extracurricular activities, and social or community activities outside the classroom. 

In conclusion, we strongly recommend that NSSE either restructure their benchmarks so 
they are more applicable to all students or at least consider excluding items from its benchmarks 
that are biased against nontraditional students. As an alternative, some benchmarks should apply 
to all types of students while others might be geared towards distinct subcategories. This type of 
specialization would reflect the growing diversity of postsecondary student populations and 
institutional efforts to serve those populations. The Carnegie Foundation’s current engagement 
in a fundamental reconsideration of the Carnegie Classification provides a model here. "We plan 
to develop a more flexible system that will permit institutions to be grouped in several ways, in 
recognition of the fact that a single classification scheme can conceal the many ways that 
institutions resemble or differ from one another," said Carnegie Senior Scholar Alexander 
McCormick (http://www.carnegiefoundation.org) . As the Foundation recognizes, one size does 
not fit all. We ask that NSSE administrators consider the same line of reasoning. 
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Introduction 

Course evaluations may contain questions for the students to describe their own gender, 
expected grade, level of study, or other information. If the survey instrument does not include 
student characteristics, administrators may gather selected student information at the section 
level from enrollment records. Faculty evaluations measure student satisfaction with the 
instructors’ teaching skills, but these reports may take on a different meaning when presented in 
conjunction with student demographics or class size. The survey manager needs to balance the 
omission of critical data on student characteristics with information overload in tasks such as 
formatting reports, carrying out special analysis, or making recommendations for survey 
instrument redesign. Gathering student characters on faculty evaluations present both problems 
and opportunities. 

Problems Gathering Student Characteristics on Course Evaluations: 

• Students may view questions about them as intrusive or irrelevant in this context 

• We gather far more data than we can analyze. Gathering student characteristics on course 
evaluations wastes both good will and class time if these data are not used. 

Opportunities for Student Characteristics on Course Evaluations: 

• It is easier to establish a clear link between student characteristics and faculty 
evaluation outcomes if the characteristics are measured on the survey, rather than 
estimated from registration records after the fact. 

Grade Inflation and Faculty Evaluations 

“Evaluations depend solely on students, and grade inflation reflects faculty worried about 
the impact students may have on their careers.” (Virginia Myers Kelly, 2005) 

Testing the conventional wisdom that students at your campus expect good grades in 
return for good faculty evaluations will be easier if your survey instrument includes a question 
about expected grades. I simulated data following the output for faculty evaluations for 
undergraduate instruction at the University at Albany-SUNY. The survey instrument for this 
simulation asks the student to provide their expected course grade in the categories: 

E/U (Failing) = 1, D = 2, C = 3, B = 4, A = 5, S = 6 
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There is also a question asking the students to rank their instructor, overall using the 
Likert scale with the following values: 

1 = Poor 2 = Fail 3 = Average 4 = Good 5 = Excellent 

Table 1 shows simulated data for undergraduate students who do not expect to fail their 
course with responses for “Instructor, Overall” and “Expected Course Grade.” We wish to test 
the null hypothesis that responses to “Instructor, Overall” are independent from responses to 
“Expected Course Grade.” If variables in the rows and columns are independent, the expected 
cell frequency for any cell in the table will equal the row total multiplied by the column total 
divided by the grand total. For example, the expected cell count for students expecting a grade 
of “D” who rated their instructors as “Poor” would be (291 * 796)/24176 = 9.58. 

Table 1 . Simulated Responses for “Instructor, Overall” and “Expected Course Grade.” 





Instructor, Overall 


Column 

Totals 


1 Poor 


2 Fair 


3 Average 


4 Good 


5 Excellent 




Expected Grade 


D 


32 


33 


59 


88 


79 


291 




C 


196 


343 


603 


998 


849 


2989 




B 


371 


622 


1436 


3872 


4765 


11066 




A 


197 


305 


781 


2773 


5774 


9830 


Row Totals 


796 


1303 


2879 


7731 


11467 


24176 



2 

The Table 1 data set should meet three main assumptions of the Chi-square (X“) test of 
independence following the guidelines set by Cochrain (1954). 

• The individual surveys represent independent trials. We assume the students did not 
collaborate on their answers when filling out the course evaluations. 

• The expected count in each category should be at least 1 . 

• No more than 1/5 of the categories should have expected counts of less than 5. 

Advantages of a Non-parametric Model in this Simulation 

Many statistical models involve the estimation of population parameters such as the mean 
or variance from a random sample. These models usually assume the probability distribution of 
the dependent variable, or variable of interest, looks like a smooth, bell-shaped curve. In 
statistical jargon, the dependent variable must have a continuous, normal distribution. The 
variable “Instructor, Overall” has a distribution that is both discontinuous and non-normal. 
Employing a non-parametric model is advantageous since no assumptions need to be made 
concerning the underlying data distribution. 
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Table 2: SPSS Output for a row by column test of independence for “Instructor, Overall” and 

“Expected Course Grade.” 





Value 


df 


Asymp. Sig. (2-sided) 


Pearson Chi-Square 
N of Valid Cases 


1488.890(a) 

24176 


12 


.000 



a 0 cells (.0%) have expected counts of less than 5. The minimum expected count is 9.58. 



Faculty ratings are not independent of expected course grade in this example (Table 2). 
Sometimes a test may show a statistically significant association between two variables that is 
irrelevant in the real world. In this example, instructors have good reason to expect lower 
student satisfaction if they assign lower grades. 



From Table 1, there is clear progression in students rating instructors as “Poor:” 

Students Expecting a grade of D: 32/291 = 11% 
Students Expecting a grade of A: 197/9830 = 2% 



There is also an important pattern for students rating instructors as “Excellent:” 
Students Expecting a grade of B: 4765/11066 = 43% 
Students Expecting a grade of A: 5774/9830 = 59% 

Policy Recommendations 



If course evaluations at your campus are similar to this simulation, several policy 
considerations may follow: 

• Faculty evaluations should be considered in conjunction with grade distributions. 

• If your institution wants to follow Harvard and fight grade inflation by setting a cap 
on “A” grades in undergraduate courses, expect lower student satisfaction ratings. 

• Expected course grade is relevant to understanding student satisfaction with 
instructors. This question should not be dropped during a survey redesign. 
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Abstract 

The widely used formulas for calculating college student retention are based on an 18-22 
year old population moving through the academy in a fall-to-fall pattern. This notion of retention 
is based on a first-time, full-time cohort. When these constraints are applied to nontraditional 
academic institutions, the results are generally misleading, largely because the enrollment 
patterns of nontraditional students do not follow a fall-to-fall pattern. Examining data from a 
college “dedicated to enabling adult students, regardless of geography or life circumstances, to 
manage and master a rigorous academic program and earn a degree,” this paper will introduce an 
alternative method for calculating retention. 1 

Relying on a first-time, full-time cohort when calculating institutional retention rates does 
not adequately represent retention for institutions serving nontraditional students. These are 
institutions whose student body consists largely of adult students engaged in part-time study. 

This paper analyzes the application of conventional approaches in unconventional settings and 
proposes an alternative method that has been developed for use at Empire State College. 

Introduction 

Historically, the convention for calculating retention among higher education institutions 
has been to identify and track the enrollment and attrition status of the first-time, full-time cohort 
of students in any given fall. This methodology is based on the traditional progression of 
educational attainment. The notion is that students engage in post- secondary study on a full-time 
basis, beginning in the fall semester immediately following high-school graduation. 

“Strongly held social norms emphasize that formal education is for the young - the first 
component in a ‘tri-partition’ of the life cycle into three orthogonal and linear segments: 
education, followed by work, followed by the leisure of old age” (Hagedorn, 2005b, p. 22). 
Research suggests an increasing number of students do not fit the profile of the traditional norm 
and are more likely to embark on alternative paths to (and through) postsecondary education 
(Choy, 2002; Han & Nesler, 2002; Horn and Carroll, 1996; Nesler & Gunnarsson, 2000). 

A special analysis published by the National Center for Education Statistics (NCES) 
reported that nearly 75% of all undergraduates are considered nontraditional to some extent 
(Choy, 2002). 2 The term nontraditional as defined by Horn & Carroll (1996), suggests the 



1 Source: Empire State College Undergraduate Catalog 2005-06. 

2 This is a special analysis included in the Condition of Education 2002 and represents the most current analysis 
concerning nontraditional students available from NCES to date. 
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presence of one or more of the following seven characteristics: delayed enrollment in post 
secondary education, part-time attendance, financial independence, full-time employment and 
simultaneous enrollment, having dependents other than a spouse, single parent status, or the 
absence of a standard high school diploma. 

Nontraditional students tend to engage in enrollment patterns and behaviors that defy 
conventional standards as well. They are more likely to stop-out, and are twice as likely to leave 
within the first year when compared to traditional-aged counterparts. It is no surprise then, that 
nontraditional students are nearly 25% less likely to obtain a degree within 5 years of enrollment 
(Horn & Carroll, 1996). 

Over the last two decades, the percentage of nontraditional students served at 4-year 
public institutions has increased from 31% in 1986 to 57.5% in 1999 (Horn & Carroll, 1996; 
Choy, 2002). Moreover, there has been an increase in the number of institutions that offer 
distance and other flexible learning options intended to serve the needs of nontraditional 
students, and this pattern is likely to continue (Hussar, 2005). 

While institutions have made strides to accommodate the arrival of nontraditional 
students, measures to assess retention and persistence have been slow to evolve. The notion of a 
“full-time, first-time, fall cohort” systematically excludes all part-time and transfer students, 
many of whom are nontraditional students. As Hagedorn (2005a) notes, “the current definitions 
and formulas do not include all students and so may provide inaccurate measure of retention” 
(Hagedom, 2005a, p. 100). 

Empire State College (ESC) is a recognized leader in distance education and is marketed 
as “the SUNY solution for working adults.” Founded in 1971, the college has 46,000 alumni and 
serves about 16,000 students annually at more than 30 locations throughout New York State. 
Over the last 5 years the number of degrees awarded annually has grown by more than 25% and 
in 2004-05, the college awarded over 2,600 undergraduate degrees. More than 90% of the 
college’s new undergraduate students enter as transfers, and two thirds of all enrolled 
undergraduates engage in part-time study. First-time, full-time enrollments represent less than 
5% of new enrollments in any given term. 

Retention rates (for ESC) identified by various sources range from 14% to 55%. The 
disparity among these statistics justifiably raised concerns, and often required explanation for 
those not familiar with the college or the characteristics of its students. Additionally, in the 
absence of consistent baseline data and appropriate methodologies, it has not been possible to 
empirically identify and broadly test factors related to retention at the college. This paper 
presents results from a comprehensive review of conventional and ‘home grown’ retention 
methodologies. 



Literature Review 

Student retention remains a major area of concern among higher education administrators 
throughout the country. Retention is often used as an indicator of institutional health and success 
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as well as a bellwether for student satisfaction. There are a number of theoretical models that 
pre-date the emergence of distance education and nontraditional students (e.g. Astin, 1977; Bean, 
1980, 1983; Tinto, 1975, 1982, 1988, 1993). The premise of these models is that there are a 
number of factors that may influence the extent to which students are retained in higher 
education or at any higher education institution. 

Perhaps one of the most influential theories has been Tinto’s student integration model 
(1975). Tinto’s model was based on a sociological approach and suggests that a combination of 
predetermined student attributes inform one’s goals; and it is the integration of one’s attributes 
and goals combined with academic and social experiences that influence the extent to which 
successful outcomes are realized or the student disengages. As Tinto explains, “it is the interplay 
between the individual’s commitment to the goal of college completion and his commitment to 
the institution that determines whether or not the individual decides to drop out” (1975, p. 96). 

In the late 1980’s, Tinto augmented his previous work to conceptualize stop-out behaviors as a 
three-stage progression: separation, transition and incorporation that relate to a student’s ability 
(or inability) to integrate into the college environment (Tinto, 1988). 

A review of literature by Kennedy and Scheckley (1999) affirms Tinto’s theory, noting 
that interactions between students and the college environment most consistently explain 
variance in persistence and attrition. Some factors may be inherent to the student, and others 
may be a function of institutional characteristics and its ability to meet the needs of different 
types of students. 

Critics of Tinto’s approach have suggested that the model fails to adequately address 
external environmental factors that can impact the extent to which a student persists in pursuit of 
higher educational goals. The notion that other factors (such as competing obligations to family 
members and financial constraints) could offset an individual’s commitment to one’s goals and 
the institution was not explored (Braxton & Hirshy, 2005; Cabrera, Castaneda, Nora and 
Hengstler, 1992; Han & Nesler, 2002; Nesler & Gunnarsson; 2000; Tinto, 1982; Swail, Redd & 
Pema, 2003). 

More recently, Eaton and Bean (1995) and later Bean and Eaton (2000) modified Tinto’s 
model using a psychological perspective. This work incorporates student attitudes (which are 
thought to influence intentions to persist) and coping abilities into Tinto’s model, suggesting that 
these additional factors contribute to one’s ability to adapt to the collegiate environment. 
“Adaptation, as measured by social and academic integration should be an attitudinal reflection 
of a student’s intention to stay or leave the institution... ultimately linked to the student’s actual 
persistence or departure” (Eaton and Bean, 1995, p. 620). 

While widely recognized and referred to among those in the field of higher education 
research, most theoretical literature does little to situate itself in the landscape of adult learners or 
more generally to nontraditional students. Early theorists could not speak to the emergence 
nontraditional post- secondary students, the motivations and the barriers they face (Kim, Collins, 
Stowe, & Chandler, 1995; Nesler & Hanner, 1998; Nesler & Gunnarsson, 2000) or the 
movement of students among various institutions. Adelman (1999) notes that, “when 60% of 
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undergraduates attend more than one institution and 40% of this group do not complete degrees, 
institutional graduation rates are not very meaningful. It is not wise to blame a college with 
superficially low graduation rates for the behavior of students who swirl through the system” 
(1999, p. ix). Adelman’s comments gain further meaning when juxtaposed with the emergence 
of institutions that are serving nontraditional populations. 

Adult students represent a growing segment of students pursuing higher education. 
According to the NCES, the proportion of older students (age 25+) pursuing postsecondary 
education has grown from 28% in 1970 to 39% in 1999 (Choy, 2002). Nearly 75% of today’s 
undergraduates are considered nontraditional to some extent, and are more likely to 
conceptualize themselves as employees first and students second (Wirt, Choy, Rooney, 
Provasnik, Sen & Tobin, 2004). Furthermore, with the growth of distance education in the 
United States, an unprecedented number of learning opportunities are now available to meet the 
needs of busy adults (Wirt, et al., 2004). 

Nontraditional students choose postsecondary enrollment for personal enrichment, skills 
acquisition and degree attainment purposes; and are more likely to experience limitations as they 
pursue postsecondary study (Hagedorn, 2005b). For example, nontraditional students typically 
engage in part-time study while simultaneously maintain commitments to competing priorities 
that limit time. This contributes to schedule conflicts, fewer course options and limited access to 
resources generally available to full-time students. Recent studies have tried to identify the best 
predictors for persistence among adult students and have determined that external factors such as 
home and personal commitments are more often not significantly related to persistence 
(Gigliotti, R. & Huff, H., 1995; Hagedom, 2005b; Kemp, 2002; Nesler & Gunnarsson, 2000). 

Given the cultural and political attachment to retention and persistence as indices of 
institutional success, it is important for audiences to recognize that nontraditional students 
represent a growing segment of postsecondary enrollments and possess individual 
characteristics, life circumstances and behaviors that distinguish them from traditionally aged 
students. Institutions dedicated to the service of this population should not be held accountable to 
the same expectations for postsecondary engagement and persistence. 

Method 

The following paragraphs document two components of this study. The first involves a 
comprehensive analysis of existing retention methodologies applied against a nontraditional 
population of students, followed by a new approach to retention that has been developed at 
Empire State College. 
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